etlplus 0.12.9__tar.gz → 0.14.1__tar.gz

{etlplus-0.12.9 → etlplus-0.14.1}/CONTRIBUTING.md

@@ -119,7 +119,7 @@ Use these guidelines to decide whether a test belongs in the unit or integration
   - Can use temporary files/directories, and stub network with fakes/mocks.
   - Examples in this repo: CLI end-to-end, pipeline smoke tests, pagination strategy, runner defaults for pagination/rate limits, target URL composition.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()`, it is integration by default.
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()`, it is integration by default.
 
 ### Where to put tests
 

{etlplus-0.12.9 → etlplus-0.14.1}/DEMO.md

@@ -196,7 +196,7 @@ $ etlplus load transformed.json file final_output.csv
 ## Demo 6: Using Python API
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract
 data = extract("file", "data.csv", format="csv")

{etlplus-0.12.9/etlplus.egg-info → etlplus-0.14.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: etlplus
-Version: 0.12.9
+Version: 0.14.1
 Summary: A Swiss Army knife for simple ETL operations
 Home-page: https://github.com/Dagitali/ETLPlus
 Author: ETLPlus Team
@@ -77,8 +77,12 @@ package and command-line interface for data extraction, validation, transformati
       - [Semi-Structured Text](#semi-structured-text)
       - [Columnar / Analytics-Friendly](#columnar--analytics-friendly)
       - [Binary Serialization and Interchange](#binary-serialization-and-interchange)
+      - [Databases and Embedded Storage](#databases-and-embedded-storage)
       - [Spreadsheets](#spreadsheets)
+      - [Statistical / Scientific / Numeric Computing](#statistical--scientific--numeric-computing)
+      - [Logs and Event Streams](#logs-and-event-streams)
       - [Data Archives](#data-archives)
+      - [Templates](#templates)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
       - [Argument Order and Required Options](#argument-order-and-required-options)
@@ -192,7 +196,7 @@ etlplus extract file examples/data/sample.csv \
 [Python API](#python-api):
 
 ```python
-from etlplus import extract, transform, validate, load
+from etlplus.ops import extract, transform, validate, load
 
 data = extract("file", "input.csv")
 ops = {"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}
@@ -219,65 +223,122 @@ DDL can be rendered from table specs for migrations or schema checks.
 
 ### Files (`file`)
 
-File formats are grouped as in `FileFormat`. Support is marked as:
+Recognized file formats are listed in the tables below. Support for reading to or writing from a recognized file format is marked as:
 
 - **Y**: implemented (may require optional dependencies)
 - **N**: stubbed or not yet implemented
 
 #### Stubbed / Placeholder
 
-| Format | Supported | Description |
-| --- | --- | --- |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
 | `stub` | N | Placeholder format for tests and future connectors. |
 
 #### Tabular & Delimited Text
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `csv` | Y | Comma-Separated Values |
-| `fwf` | N | Fixed-Width Fields |
-| `dat` | N | Generic data file, often delimited or fixed-width |
-| `psv` | N | Pipe-Separated Values |
-| `tab` | N | Often synonymous with TSV |
-| `tsv` | Y | Tab-Separated Values |
-| `txt` | Y | Plain text, often delimited or fixed-width |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `csv` | Y | Y | Comma-Separated Values |
+| `dat` | N | N | Generic data file, often delimited or fixed-width |
+| `fwf` | N | N | Fixed-Width Fields |
+| `psv` | N | N | Pipe-Separated Values |
+| `tab` | N | N | Often synonymous with TSV |
+| `tsv` | Y | Y | Tab-Separated Values |
+| `txt` | Y | Y | Plain text, often delimited or fixed-width |
 
 #### Semi-Structured Text
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `json` | Y | JavaScript Object Notation |
-| `ndjson` | Y | Newline-Delimited JSON |
-| `xml` | Y | Extensible Markup Language |
-| `yaml` | Y | YAML Ain't Markup Language |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `cfg` | N | N | Config-style key-value pairs |
+| `conf` | N | N | Config-style key-value pairs |
+| `ini` | N | N | Config-style key-value pairs |
+| `json` | Y | Y | JavaScript Object Notation |
+| `ndjson` | Y | Y | Newline-Delimited JSON |
+| `properties` | N | N | Java-style key-value pairs |
+| `toml` | N | N | Tom's Obvious Minimal Language |
+| `xml` | Y | Y | Extensible Markup Language |
+| `yaml` | Y | Y | YAML Ain't Markup Language |
 
 #### Columnar / Analytics-Friendly
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `feather` | Y | Apache Arrow Feather |
-| `orc` | Y | Optimized Row Columnar; common in Hadoop |
-| `parquet` | Y | Apache Parquet; common in Big Data |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `arrow` | N | N | Apache Arrow IPC |
+| `feather` | Y | Y | Apache Arrow Feather |
+| `orc` | Y | Y | Optimized Row Columnar; common in Hadoop |
+| `parquet` | Y | Y | Apache Parquet; common in Big Data |
 
 #### Binary Serialization and Interchange
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `avro` | Y | Apache Avro |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `avro` | Y | Y | Apache Avro |
+| `bson` | N | N | Binary JSON; common with MongoDB exports/dumps |
+| `cbor` | N | N | Concise Binary Object Representation |
+| `ion` | N | N | Amazon Ion |
+| `msgpack` | N | N | MessagePack |
+| `pb` | N | N | Protocol Buffers (Google Protobuf) |
+| `pbf` | N | N | Protocolbuffer Binary Format; often for GIS data |
+| `proto` | N | N | Protocol Buffers schema; often in .pb / .bin |
+
+#### Databases and Embedded Storage
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `accdb` | N | N | Microsoft Access (newer format) |
+| `duckdb` | N | N | DuckDB |
+| `mdb` | N | N | Microsoft Access (older format) |
+| `sqlite` | N | N | SQLite |
 
 #### Spreadsheets
 
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `numbers` | N | N | Apple Numbers |
+| `ods` | N | N | OpenDocument |
+| `wks` | N | N | Lotus 1-2-3  |
+| `xls` | Y | Y | Microsoft Excel (BIFF) |
+| `xlsm` | N | N | Microsoft Excel Macro-Enabled (Open XML) |
+| `xlsx` | Y | Y | Microsoft Excel (Open XML) |
+
+#### Statistical / Scientific / Numeric Computing
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `dta` | N | N | Stata |
+| `hdf5` | N | N | Hierarchical Data Format |
+| `mat` | N | N | MATLAB |
+| `nc` | N | N | NetCDF |
+| `rda` | N | N | RData workspace/object |
+| `rds` | N | N | R data |
+| `sas7bdat` | N | N | SAS data |
+| `sav` | N | N | SPSS data |
+| `sylk` | N | N | Symbolic Link |
+| `xpt` | N | N | SAS Transport |
+| `zsav` | N | N | Compressed SPSS data |
+
+#### Logs and Event Streams
+
 | Format | Supported | Description |
 | --- | --- | --- |
-| `xls` | Y | Microsoft Excel (BIFF); read-only |
-| `xlsx` | Y | Microsoft Excel (Open XML) |
+| `log` | N | N | Generic log file |
 
 #### Data Archives
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `gz` | Y | Gzip-compressed file |
-| `zip` | Y | ZIP archive |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `gz` | Y | Y | Gzip-compressed file |
+| `zip` | Y | Y | ZIP archive |
+
+#### Templates
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `hbs` | N | N | Handlebars |
+| `jinja2` | N | N | Jinja2 |
+| `mustache` | N | N | Mustache |
+| `vm` | N | N | Apache Velocity |
 
 ## Usage
 
@@ -470,7 +531,7 @@ cat examples/data/sample.json \
 Use ETLPlus as a Python library:
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract data
 data = extract("file", "data.json")
@@ -665,7 +726,7 @@ We split tests into two layers:
   pagination + rate limit defaults, file/API connector interactions) may touch temp files and use
   fake clients.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()` it’s integration by default.  Full
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()` it’s integration by default. Full
 criteria: [`CONTRIBUTING.md#testing`](CONTRIBUTING.md#testing).
 
 ### Code Coverage

{etlplus-0.12.9 → etlplus-0.14.1}/README.md

@@ -27,8 +27,12 @@ package and command-line interface for data extraction, validation, transformati
       - [Semi-Structured Text](#semi-structured-text)
       - [Columnar / Analytics-Friendly](#columnar--analytics-friendly)
       - [Binary Serialization and Interchange](#binary-serialization-and-interchange)
+      - [Databases and Embedded Storage](#databases-and-embedded-storage)
       - [Spreadsheets](#spreadsheets)
+      - [Statistical / Scientific / Numeric Computing](#statistical--scientific--numeric-computing)
+      - [Logs and Event Streams](#logs-and-event-streams)
       - [Data Archives](#data-archives)
+      - [Templates](#templates)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
       - [Argument Order and Required Options](#argument-order-and-required-options)
@@ -142,7 +146,7 @@ etlplus extract file examples/data/sample.csv \
 [Python API](#python-api):
 
 ```python
-from etlplus import extract, transform, validate, load
+from etlplus.ops import extract, transform, validate, load
 
 data = extract("file", "input.csv")
 ops = {"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}
@@ -169,65 +173,122 @@ DDL can be rendered from table specs for migrations or schema checks.
 
 ### Files (`file`)
 
-File formats are grouped as in `FileFormat`. Support is marked as:
+Recognized file formats are listed in the tables below. Support for reading to or writing from a recognized file format is marked as:
 
 - **Y**: implemented (may require optional dependencies)
 - **N**: stubbed or not yet implemented
 
 #### Stubbed / Placeholder
 
-| Format | Supported | Description |
-| --- | --- | --- |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
 | `stub` | N | Placeholder format for tests and future connectors. |
 
 #### Tabular & Delimited Text
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `csv` | Y | Comma-Separated Values |
-| `fwf` | N | Fixed-Width Fields |
-| `dat` | N | Generic data file, often delimited or fixed-width |
-| `psv` | N | Pipe-Separated Values |
-| `tab` | N | Often synonymous with TSV |
-| `tsv` | Y | Tab-Separated Values |
-| `txt` | Y | Plain text, often delimited or fixed-width |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `csv` | Y | Y | Comma-Separated Values |
+| `dat` | N | N | Generic data file, often delimited or fixed-width |
+| `fwf` | N | N | Fixed-Width Fields |
+| `psv` | N | N | Pipe-Separated Values |
+| `tab` | N | N | Often synonymous with TSV |
+| `tsv` | Y | Y | Tab-Separated Values |
+| `txt` | Y | Y | Plain text, often delimited or fixed-width |
 
 #### Semi-Structured Text
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `json` | Y | JavaScript Object Notation |
-| `ndjson` | Y | Newline-Delimited JSON |
-| `xml` | Y | Extensible Markup Language |
-| `yaml` | Y | YAML Ain't Markup Language |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `cfg` | N | N | Config-style key-value pairs |
+| `conf` | N | N | Config-style key-value pairs |
+| `ini` | N | N | Config-style key-value pairs |
+| `json` | Y | Y | JavaScript Object Notation |
+| `ndjson` | Y | Y | Newline-Delimited JSON |
+| `properties` | N | N | Java-style key-value pairs |
+| `toml` | N | N | Tom's Obvious Minimal Language |
+| `xml` | Y | Y | Extensible Markup Language |
+| `yaml` | Y | Y | YAML Ain't Markup Language |
 
 #### Columnar / Analytics-Friendly
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `feather` | Y | Apache Arrow Feather |
-| `orc` | Y | Optimized Row Columnar; common in Hadoop |
-| `parquet` | Y | Apache Parquet; common in Big Data |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `arrow` | N | N | Apache Arrow IPC |
+| `feather` | Y | Y | Apache Arrow Feather |
+| `orc` | Y | Y | Optimized Row Columnar; common in Hadoop |
+| `parquet` | Y | Y | Apache Parquet; common in Big Data |
 
 #### Binary Serialization and Interchange
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `avro` | Y | Apache Avro |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `avro` | Y | Y | Apache Avro |
+| `bson` | N | N | Binary JSON; common with MongoDB exports/dumps |
+| `cbor` | N | N | Concise Binary Object Representation |
+| `ion` | N | N | Amazon Ion |
+| `msgpack` | N | N | MessagePack |
+| `pb` | N | N | Protocol Buffers (Google Protobuf) |
+| `pbf` | N | N | Protocolbuffer Binary Format; often for GIS data |
+| `proto` | N | N | Protocol Buffers schema; often in .pb / .bin |
+
+#### Databases and Embedded Storage
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `accdb` | N | N | Microsoft Access (newer format) |
+| `duckdb` | N | N | DuckDB |
+| `mdb` | N | N | Microsoft Access (older format) |
+| `sqlite` | N | N | SQLite |
 
 #### Spreadsheets
 
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `numbers` | N | N | Apple Numbers |
+| `ods` | N | N | OpenDocument |
+| `wks` | N | N | Lotus 1-2-3  |
+| `xls` | Y | Y | Microsoft Excel (BIFF) |
+| `xlsm` | N | N | Microsoft Excel Macro-Enabled (Open XML) |
+| `xlsx` | Y | Y | Microsoft Excel (Open XML) |
+
+#### Statistical / Scientific / Numeric Computing
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `dta` | N | N | Stata |
+| `hdf5` | N | N | Hierarchical Data Format |
+| `mat` | N | N | MATLAB |
+| `nc` | N | N | NetCDF |
+| `rda` | N | N | RData workspace/object |
+| `rds` | N | N | R data |
+| `sas7bdat` | N | N | SAS data |
+| `sav` | N | N | SPSS data |
+| `sylk` | N | N | Symbolic Link |
+| `xpt` | N | N | SAS Transport |
+| `zsav` | N | N | Compressed SPSS data |
+
+#### Logs and Event Streams
+
 | Format | Supported | Description |
 | --- | --- | --- |
-| `xls` | Y | Microsoft Excel (BIFF); read-only |
-| `xlsx` | Y | Microsoft Excel (Open XML) |
+| `log` | N | N | Generic log file |
 
 #### Data Archives
 
-| Format | Supported | Description |
-| --- | --- | --- |
-| `gz` | Y | Gzip-compressed file |
-| `zip` | Y | ZIP archive |
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `gz` | Y | Y | Gzip-compressed file |
+| `zip` | Y | Y | ZIP archive |
+
+#### Templates
+
+| Format | Read | Write | Description |
+| --- | --- | --- | --- |
+| `hbs` | N | N | Handlebars |
+| `jinja2` | N | N | Jinja2 |
+| `mustache` | N | N | Mustache |
+| `vm` | N | N | Apache Velocity |
 
 ## Usage
 
@@ -420,7 +481,7 @@ cat examples/data/sample.json \
 Use ETLPlus as a Python library:
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract data
 data = extract("file", "data.json")
@@ -615,7 +676,7 @@ We split tests into two layers:
   pagination + rate limit defaults, file/API connector interactions) may touch temp files and use
   fake clients.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()` it’s integration by default.  Full
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()` it’s integration by default. Full
 criteria: [`CONTRIBUTING.md#testing`](CONTRIBUTING.md#testing).
 
 ### Code Coverage

{etlplus-0.12.9 → etlplus-0.14.1}/docs/pipeline-guide.md

@@ -281,7 +281,7 @@ section.
 
 ## Validations
 
-Validation rule sets map field names to rules, mirroring `etlplus.validate.FieldRules`:
+Validation rule sets map field names to rules, mirroring `etlplus.ops.validate.FieldRules`:
 
 ```yaml
 validations:
@@ -297,7 +297,7 @@ validations:
 
 ## Transforms
 
-Transformation pipelines follow `etlplus.transform` shapes exactly:
+Transformation pipelines follow `etlplus.ops.transform` shapes exactly:
 
 ```yaml
 transforms:
@@ -390,10 +390,20 @@ target:
 ```yaml
 jobs:
   - name: file_to_file_customers
+    depends_on: [seed_customers]
     extract: { source: customers_csv }
     validate: { ruleset: customers_basic }
     transform: { pipeline: clean_customers }
     load: { target: customers_json_out }
+  - name: seed_customers
+    extract: { source: seed_customers_csv }
+    load: { target: customers_db_out }
+```
+
+Notes:
+
+- `depends_on` is optional and can be a string or list of job names.
+- Jobs without dependencies run first when ordered as a DAG.
 
 ## Running pipelines (CLI and Python)
 
@@ -421,14 +431,14 @@ Notes:
 - Environment-variable substitution (e.g. `${GITHUB_TOKEN}`) is applied the same way as when loading
   configs via the Python API.
 - For more details on the orchestration implementation, see
-  [Runner internals: etlplus.run](run-module.md).
+  [Runner internals: etlplus.ops.run](run-module.md).
 
-### Python: `etlplus.run.run`
+### Python: `etlplus.ops.run.run`
 
 To trigger a job programmatically, use the high-level runner function exposed by the package:
 
 ```python
-from etlplus.run import run as run_job
+from etlplus.ops.run import run as run_job
 
 result = run_job(
     job="file_to_file_customers",

{etlplus-0.12.9 → etlplus-0.14.1}/etlplus/README.md

@@ -23,7 +23,7 @@ Back to project overview: see the top-level [README](../README.md).
 ## Quickstart
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 data = extract("file", "input.csv")
 filtered = transform(data, {"filter": {"field": "age", "op": "gt", "value": 25}})

etlplus-0.14.1/etlplus/__init__.py

@@ -0,0 +1,18 @@
+"""
+:mod:`etlplus` package.
+
+Top-level facade for the ETLPlus toolkit.
+"""
+
+from .__version__ import __version__
+
+__author__ = 'ETLPlus Team'
+
+
+# SECTION: EXPORTS ========================================================== #
+
+
+__all__ = [
+    '__author__',
+    '__version__',
+]

{etlplus-0.12.9 → etlplus-0.14.1}/etlplus/api/__init__.py

@@ -98,6 +98,10 @@ from .types import Headers
 from .types import Params
 from .types import RequestOptions
 from .types import Url
+from .utils import compose_api_request_env
+from .utils import compose_api_target_env
+from .utils import paginate_with_client
+from .utils import resolve_request
 
 # SECTION: EXPORTS ========================================================== #
 
@@ -122,6 +126,10 @@ __all__ = [
     'PaginationType',
     # Functions
     'build_http_adapter',
+    'compose_api_request_env',
+    'compose_api_target_env',
+    'paginate_with_client',
+    'resolve_request',
     # Type Aliases
     'CursorPaginationConfigMap',
     'Headers',

{etlplus-0.12.9 → etlplus-0.14.1}/etlplus/api/endpoint_client.py

@@ -455,7 +455,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.get(url, **kwargs)
 
@@ -479,7 +479,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.post(url, **kwargs)
 
@@ -506,7 +506,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.request(method, url, **kwargs)