etlplus 0.4.1__tar.gz → 0.5.5__tar.gz

{etlplus-0.4.1 → etlplus-0.5.5}/.pre-commit-config.yaml

@@ -159,7 +159,11 @@ repos:
     rev: v1.19.0
     hooks:
       - id: mypy
-        args: [--ignore-missing-imports, --install-types, --non-interactive]
+        args:
+          - --cache-dir=.mypy_cache/pre-commit
+          - --ignore-missing-imports
+          - --install-types
+          - --non-interactive
 
   - repo: https://github.com/pycqa/flake8
     rev: 7.3.0

{etlplus-0.4.1 → etlplus-0.5.5}/DEMO.md

@@ -58,7 +58,7 @@ John Doe,30,New York
 Jane Smith,25,Los Angeles
 CSVDATA
 
-$ etlplus extract file users.csv --format csv
+$ etlplus extract users.csv
 [
   {
     "name": "John Doe",
@@ -151,7 +151,7 @@ $ etlplus load '{"name": "John", "status": "active"}' file output.json
 $ etlplus load '[
     {"name": "John", "email": "john@example.com"},
     {"name": "Jane", "email": "jane@example.com"}
-  ]' file users.csv --format csv
+  ]' --to users.csv
 {
   "status": "success",
   "message": "Data loaded to users.csv",
@@ -170,14 +170,14 @@ This example shows a complete ETL workflow:
 
 ```bash
 # Step 1: Extract
-$ etlplus extract file raw_data.csv --format csv -o extracted.json
+$ etlplus extract raw_data.csv > extracted.json
 
 # Step 2: Transform
-$ etlplus transform extracted.json \
+$ etlplus transform --from extracted.json \
   --operations '{
     "filter": {"field": "age", "op": "gte", "value": 18},
     "select": ["name", "email", "age"]
-  }' -o transformed.json
+  }' --to transformed.json
 
 # Step 3: Validate
 $ etlplus validate transformed.json \

etlplus-0.5.5/MANIFEST.in

@@ -0,0 +1,12 @@
+# MANIFEST.in
+# ETLPlus
+#
+# Copyright © 2026 Dagitali LLC. All rights reserved.
+#
+# Contains commands that allow lists of files to be discovered and manipulated.
+#
+# See:
+# 1. https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
+
+# Include Jinja template files in the etlplus package
+recursive-include etlplus/templates *.j2

{etlplus-0.4.1/etlplus.egg-info → etlplus-0.5.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: etlplus
-Version: 0.4.1
+Version: 0.5.5
 Summary: A Swiss Army knife for simple ETL operations
 Home-page: https://github.com/Dagitali/ETLPlus
 Author: ETLPlus Team
@@ -61,13 +61,15 @@ package and command-line interface for data extraction, validation, transformati
   - [Quickstart](#quickstart)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
+      - [Inspect Pipelines](#inspect-pipelines)
+      - [Render SQL DDL](#render-sql-ddl)
       - [Extract Data](#extract-data)
       - [Validate Data](#validate-data)
       - [Transform Data](#transform-data)
       - [Load Data](#load-data)
     - [Python API](#python-api)
     - [Complete ETL Pipeline Example](#complete-etl-pipeline-example)
-    - [Environment Variables](#environment-variables)
+    - [Format Overrides](#format-overrides)
   - [Transformation Operations](#transformation-operations)
     - [Filter Operations](#filter-operations)
     - [Aggregation Functions](#aggregation-functions)
@@ -79,6 +81,8 @@ package and command-line interface for data extraction, validation, transformati
       - [Test Layers](#test-layers)
     - [Code Coverage](#code-coverage)
     - [Linting](#linting)
+    - [Updating Demo Snippets](#updating-demo-snippets)
+    - [Releasing to PyPI](#releasing-to-pypi)
   - [Links](#links)
   - [License](#license)
   - [Contributing](#contributing)
@@ -86,6 +90,14 @@ package and command-line interface for data extraction, validation, transformati
 
 ## Features
 
+- **Check** data pipeline definitions before running them:
+  - Summarize jobs, sources, targets, and transforms
+  - Confirm configuration changes by printing focused sections on demand
+
+- **Render** SQL DDL from shared table specs:
+  - Generate CREATE TABLE or view statements
+  - Swap templates or direct output to files for database migrations
+
 - **Extract** data from multiple sources:
   - Files (CSV, JSON, XML, YAML)
   - Databases (connection string support)
@@ -167,11 +179,49 @@ etlplus --help
 etlplus --version
 ```
 
+#### Check Pipelines
+
+Use `etlplus check` to explore pipeline YAML definitions without running them. The command can print
+job names, summarize configured sources and targets, or drill into specific sections.
+
+List jobs and show a pipeline summary:
+```bash
+etlplus check --config examples/configs/pipeline.yml --jobs
+etlplus check --config examples/configs/pipeline.yml --summary
+```
+
+Show sources or transforms for troubleshooting:
+```bash
+etlplus check --config examples/configs/pipeline.yml --sources
+etlplus check --config examples/configs/pipeline.yml --transforms
+```
+
+#### Render SQL DDL
+
+Use `etlplus render` to turn table schema specs into ready-to-run SQL. Render from a pipeline config
+or from a standalone schema file, and choose the built-in `ddl` or `view` templates (or provide your
+own).
+
+Render all tables defined in a pipeline:
+```bash
+etlplus render --config examples/configs/pipeline.yml --template ddl
+```
+
+Render a single table in that pipeline:
+```bash
+etlplus render --config examples/configs/pipeline.yml --table customers --template view
+```
+
+Render from a standalone table spec to a file:
+```bash
+etlplus render --spec schemas/customer.yml --template view -o temp/customer_view.sql
+```
+
 #### Extract Data
 
-Note: For file sources, the format is inferred from the filename extension; the `--format` option is
-ignored.  To treat passing `--format` as an error for file sources, either set
-`ETLPLUS_FORMAT_BEHAVIOR=error` or pass the CLI flag `--strict-format`.
+Note: For file sources, the format is normally inferred from the filename extension. Use
+`--source-format` to override inference when a file lacks an extension or when you want to force a
+specific parser.
 
 Extract from JSON file:
 ```bash
@@ -212,6 +262,20 @@ etlplus validate examples/data/sample.json --rules '{"email": {"type": "string",
 
 #### Transform Data
 
+When piping data through `etlplus transform`, use `--source-format` whenever the SOURCE argument is
+`-` or a literal payload, mirroring the `etlplus extract` semantics. Use `--target-format` to
+control the emitted format for stdout or other non-file outputs, just like `etlplus load`. File
+paths continue to infer formats from their extensions. Use `--from` to override the inferred source
+connector type and `--to` to override the inferred target connector type, matching the `etlplus
+extract`/`etlplus load` behavior.
+
+Transform file inputs while overriding connector types:
+```bash
+etlplus transform --from file examples/data/sample.json \
+  --operations '{"select": ["name", "email"]}' \
+  --to file -o temp/selected_output.json
+```
+
 Filter and select fields:
 ```bash
 etlplus transform '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]' \
@@ -235,19 +299,24 @@ etlplus transform examples/data/sample.json --operations '{"map": {"name": "new_
 
 #### Load Data
 
+`etlplus load` consumes JSON from stdin; provide only the target argument plus optional flags.
+
 Load to JSON file:
 ```bash
-etlplus load '{"name": "John", "age": 30}' file temp/sample_output.json
+etlplus extract file examples/data/sample.json \
+  | etlplus load --to file temp/sample_output.json
 ```
 
 Load to CSV file:
 ```bash
-etlplus load '[{"name": "John", "age": 30}]' file temp/sample_output.csv
+etlplus extract file examples/data/sample.csv \
+  | etlplus load --to file temp/sample_output.csv
 ```
 
 Load to REST API:
 ```bash
-etlplus load examples/data/sample.json api https://api.example.com/endpoint
+cat examples/data/sample.json \
+  | etlplus load --to api https://api.example.com/endpoint
 ```
 
 ### Python API
@@ -285,6 +354,19 @@ For YAML-driven pipelines executed end-to-end (extract → validate → transfor
 - Authoring: [`docs/pipeline-guide.md`](docs/pipeline-guide.md)
 - Runner API and internals: [`docs/run-module.md`](docs/run-module.md)
 
+CLI quick reference for pipelines:
+
+```bash
+# List jobs or show a pipeline summary
+etlplus check --config examples/configs/pipeline.yml --jobs
+etlplus check --config examples/configs/pipeline.yml --summary
+
+# Run a job
+etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers
+
+# Deprecated shim (will be removed): etlplus pipeline
+```
+
 ### Complete ETL Pipeline Example
 
 ```bash
@@ -301,41 +383,28 @@ etlplus validate temp/sample_transformed.json \
   --rules '{"name": {"type": "string", "required": true}, "email": {"type": "string", "required": true}}'
 
 # 4. Load to CSV
-etlplus load temp/sample_transformed.json file temp/sample_output.csv
+cat temp/sample_transformed.json \
+  | etlplus load --to temp/sample_output.csv
 ```
 
-### Environment Variables
-
-ETLPlus honors a small number of environment toggles to refine CLI behavior:
+### Format Overrides
 
-- `ETLPLUS_FORMAT_BEHAVIOR`: controls what happens when `--format` is provided for
-  file sources or targets (extract/load) where the format is inferred from the
-  filename extension.
-  - `error|fail|strict`: treat as error (non-zero exit)
-  - `warn` (default): print a warning to stderr
-  - `ignore|silent`: no message
-- Precedence: the CLI flag `--strict-format` overrides the environment.
+`--source-format` and `--target-format` override whichever format would normally be inferred from a
+file extension. This is useful when an input lacks an extension (for example, `records.txt` that
+actually contains CSV) or when you intentionally want to treat a file as another format.
 
 Examples (zsh):
 
 ```zsh
-# Warn (default)
-etlplus extract file data.csv --format csv
-etlplus load data.json file out.csv --format csv
-
-# Enforce error via environment
-ETLPLUS_FORMAT_BEHAVIOR=error \
-  etlplus extract file data.csv --format csv
-ETLPLUS_FORMAT_BEHAVIOR=error \
-  etlplus load data.json file out.csv --format csv
-
-# Equivalent strict behavior via flag (overrides environment)
-etlplus extract file data.csv --format csv --strict-format
-etlplus load data.json file out.csv --format csv --strict-format
-
-# Recommended: rely on extension, no --format needed for files
-etlplus extract file data.csv
-etlplus load data.json file out.csv
+# Force CSV parsing for an extension-less file
+etlplus extract --from file data.txt --source-format csv
+
+# Write CSV to a file without the .csv suffix
+etlplus load --to file output.bin --target-format csv < data.json
+
+# Leave the flags off when extensions already match the desired format
+etlplus extract --from file data.csv
+etlplus load --to file data.json < data.json
 ```
 
 ## Transformation Operations

{etlplus-0.4.1 → etlplus-0.5.5}/README.md

@@ -19,13 +19,15 @@ package and command-line interface for data extraction, validation, transformati
   - [Quickstart](#quickstart)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
+      - [Inspect Pipelines](#inspect-pipelines)
+      - [Render SQL DDL](#render-sql-ddl)
       - [Extract Data](#extract-data)
       - [Validate Data](#validate-data)
       - [Transform Data](#transform-data)
       - [Load Data](#load-data)
     - [Python API](#python-api)
     - [Complete ETL Pipeline Example](#complete-etl-pipeline-example)
-    - [Environment Variables](#environment-variables)
+    - [Format Overrides](#format-overrides)
   - [Transformation Operations](#transformation-operations)
     - [Filter Operations](#filter-operations)
     - [Aggregation Functions](#aggregation-functions)
@@ -37,6 +39,8 @@ package and command-line interface for data extraction, validation, transformati
       - [Test Layers](#test-layers)
     - [Code Coverage](#code-coverage)
     - [Linting](#linting)
+    - [Updating Demo Snippets](#updating-demo-snippets)
+    - [Releasing to PyPI](#releasing-to-pypi)
   - [Links](#links)
   - [License](#license)
   - [Contributing](#contributing)
@@ -44,6 +48,14 @@ package and command-line interface for data extraction, validation, transformati
 
 ## Features
 
+- **Check** data pipeline definitions before running them:
+  - Summarize jobs, sources, targets, and transforms
+  - Confirm configuration changes by printing focused sections on demand
+
+- **Render** SQL DDL from shared table specs:
+  - Generate CREATE TABLE or view statements
+  - Swap templates or direct output to files for database migrations
+
 - **Extract** data from multiple sources:
   - Files (CSV, JSON, XML, YAML)
   - Databases (connection string support)
@@ -125,11 +137,49 @@ etlplus --help
 etlplus --version
 ```
 
+#### Check Pipelines
+
+Use `etlplus check` to explore pipeline YAML definitions without running them. The command can print
+job names, summarize configured sources and targets, or drill into specific sections.
+
+List jobs and show a pipeline summary:
+```bash
+etlplus check --config examples/configs/pipeline.yml --jobs
+etlplus check --config examples/configs/pipeline.yml --summary
+```
+
+Show sources or transforms for troubleshooting:
+```bash
+etlplus check --config examples/configs/pipeline.yml --sources
+etlplus check --config examples/configs/pipeline.yml --transforms
+```
+
+#### Render SQL DDL
+
+Use `etlplus render` to turn table schema specs into ready-to-run SQL. Render from a pipeline config
+or from a standalone schema file, and choose the built-in `ddl` or `view` templates (or provide your
+own).
+
+Render all tables defined in a pipeline:
+```bash
+etlplus render --config examples/configs/pipeline.yml --template ddl
+```
+
+Render a single table in that pipeline:
+```bash
+etlplus render --config examples/configs/pipeline.yml --table customers --template view
+```
+
+Render from a standalone table spec to a file:
+```bash
+etlplus render --spec schemas/customer.yml --template view -o temp/customer_view.sql
+```
+
 #### Extract Data
 
-Note: For file sources, the format is inferred from the filename extension; the `--format` option is
-ignored.  To treat passing `--format` as an error for file sources, either set
-`ETLPLUS_FORMAT_BEHAVIOR=error` or pass the CLI flag `--strict-format`.
+Note: For file sources, the format is normally inferred from the filename extension. Use
+`--source-format` to override inference when a file lacks an extension or when you want to force a
+specific parser.
 
 Extract from JSON file:
 ```bash
@@ -170,6 +220,20 @@ etlplus validate examples/data/sample.json --rules '{"email": {"type": "string",
 
 #### Transform Data
 
+When piping data through `etlplus transform`, use `--source-format` whenever the SOURCE argument is
+`-` or a literal payload, mirroring the `etlplus extract` semantics. Use `--target-format` to
+control the emitted format for stdout or other non-file outputs, just like `etlplus load`. File
+paths continue to infer formats from their extensions. Use `--from` to override the inferred source
+connector type and `--to` to override the inferred target connector type, matching the `etlplus
+extract`/`etlplus load` behavior.
+
+Transform file inputs while overriding connector types:
+```bash
+etlplus transform --from file examples/data/sample.json \
+  --operations '{"select": ["name", "email"]}' \
+  --to file -o temp/selected_output.json
+```
+
 Filter and select fields:
 ```bash
 etlplus transform '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]' \
@@ -193,19 +257,24 @@ etlplus transform examples/data/sample.json --operations '{"map": {"name": "new_
 
 #### Load Data
 
+`etlplus load` consumes JSON from stdin; provide only the target argument plus optional flags.
+
 Load to JSON file:
 ```bash
-etlplus load '{"name": "John", "age": 30}' file temp/sample_output.json
+etlplus extract file examples/data/sample.json \
+  | etlplus load --to file temp/sample_output.json
 ```
 
 Load to CSV file:
 ```bash
-etlplus load '[{"name": "John", "age": 30}]' file temp/sample_output.csv
+etlplus extract file examples/data/sample.csv \
+  | etlplus load --to file temp/sample_output.csv
 ```
 
 Load to REST API:
 ```bash
-etlplus load examples/data/sample.json api https://api.example.com/endpoint
+cat examples/data/sample.json \
+  | etlplus load --to api https://api.example.com/endpoint
 ```
 
 ### Python API
@@ -243,6 +312,19 @@ For YAML-driven pipelines executed end-to-end (extract → validate → transfor
 - Authoring: [`docs/pipeline-guide.md`](docs/pipeline-guide.md)
 - Runner API and internals: [`docs/run-module.md`](docs/run-module.md)
 
+CLI quick reference for pipelines:
+
+```bash
+# List jobs or show a pipeline summary
+etlplus check --config examples/configs/pipeline.yml --jobs
+etlplus check --config examples/configs/pipeline.yml --summary
+
+# Run a job
+etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers
+
+# Deprecated shim (will be removed): etlplus pipeline
+```
+
 ### Complete ETL Pipeline Example
 
 ```bash
@@ -259,41 +341,28 @@ etlplus validate temp/sample_transformed.json \
   --rules '{"name": {"type": "string", "required": true}, "email": {"type": "string", "required": true}}'
 
 # 4. Load to CSV
-etlplus load temp/sample_transformed.json file temp/sample_output.csv
+cat temp/sample_transformed.json \
+  | etlplus load --to temp/sample_output.csv
 ```
 
-### Environment Variables
-
-ETLPlus honors a small number of environment toggles to refine CLI behavior:
+### Format Overrides
 
-- `ETLPLUS_FORMAT_BEHAVIOR`: controls what happens when `--format` is provided for
-  file sources or targets (extract/load) where the format is inferred from the
-  filename extension.
-  - `error|fail|strict`: treat as error (non-zero exit)
-  - `warn` (default): print a warning to stderr
-  - `ignore|silent`: no message
-- Precedence: the CLI flag `--strict-format` overrides the environment.
+`--source-format` and `--target-format` override whichever format would normally be inferred from a
+file extension. This is useful when an input lacks an extension (for example, `records.txt` that
+actually contains CSV) or when you intentionally want to treat a file as another format.
 
 Examples (zsh):
 
 ```zsh
-# Warn (default)
-etlplus extract file data.csv --format csv
-etlplus load data.json file out.csv --format csv
-
-# Enforce error via environment
-ETLPLUS_FORMAT_BEHAVIOR=error \
-  etlplus extract file data.csv --format csv
-ETLPLUS_FORMAT_BEHAVIOR=error \
-  etlplus load data.json file out.csv --format csv
-
-# Equivalent strict behavior via flag (overrides environment)
-etlplus extract file data.csv --format csv --strict-format
-etlplus load data.json file out.csv --format csv --strict-format
-
-# Recommended: rely on extension, no --format needed for files
-etlplus extract file data.csv
-etlplus load data.json file out.csv
+# Force CSV parsing for an extension-less file
+etlplus extract --from file data.txt --source-format csv
+
+# Write CSV to a file without the .csv suffix
+etlplus load --to file output.bin --target-format csv < data.json
+
+# Leave the flags off when extensions already match the desired format
+etlplus extract --from file data.csv
+etlplus load --to file data.json < data.json
 ```
 
 ## Transformation Operations

{etlplus-0.4.1 → etlplus-0.5.5}/docs/pipeline-guide.md

@@ -245,13 +245,13 @@ job. Those values are merged into the client configuration and forwarded to
 `EndpointClient.paginate(..., rate_limit_overrides=...)`, ensuring only that job’s paginator is sped
 up or slowed down.
 
-Environment / format inference note:
+Format override note:
 
-When extracting from file sources, ETLPlus infers the format from the filename extension (e.g.
-`.csv`, `.json`, `.xml`, `.yaml`). Passing an explicit CLI `--format` for files is ignored unless
-strict mode is enabled. To enforce an error if contributors still specify a redundant `--format`
-flag in scripts or CI runners, set the environment variable `ETLPLUS_FORMAT_BEHAVIOR=error` or use
-the CLI flag `--strict-format`. This keeps pipelines cleaner by relying on naming conventions.
+When extracting from file sources, ETLPlus still infers the format from the filename extension
+(`.csv`, `.json`, `.xml`, `.yaml`). However, `--source-format` and `--target-format` now override
+that inference for both Typer- and argparse-based CLIs. This means you can safely point at files
+without/extensions or with misleading suffixes and force the desired parser or writer without having
+to rename the file first.
 
 Note: When using a service + endpoint in a source, URL composition (including `base_path`) is
 handled automatically. See “Runner behavior with base_path (sources and targets)” in the APIs
@@ -378,31 +378,32 @@ jobs:
 Once you have a pipeline YAML, you can run jobs either from the
 command line or directly from Python.
 
-### CLI: `etlplus pipeline` and `etlplus run`
+### CLI: `etlplus check` (inspect) and `etlplus run` (execute)
 
-List jobs defined in a pipeline file:
+List jobs or show a summary from a pipeline file:
 
 ```bash
-etlplus pipeline --config examples/configs/pipeline.yml --list
+etlplus check --config examples/configs/pipeline.yml --jobs
+etlplus check --config examples/configs/pipeline.yml --summary
 ```
 
 Run a specific job end-to-end (extract → validate → transform → load):
 
 ```bash
-etlplus pipeline --config examples/configs/pipeline.yml --run file_to_file_customers
-
-# Equivalent, using the dedicated run command
 etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers
 ```
 
 Notes:
 
-- Both commands read the same YAML schema described in this guide.
+- These commands read the same YAML schema described in this guide.
 - Environment-variable substitution (e.g. `${GITHUB_TOKEN}`) is applied the same way as when loading
   configs via the Python API.
 - For more details on the orchestration implementation, see
   [Runner internals: etlplus.run](run-module.md).
 
+Deprecated: `etlplus pipeline` is still available as a shim but will be removed in a future release;
+prefer `check` and `run`.
+
 ### Python: `etlplus.run.run`
 
 To trigger a job programmatically, use the high-level runner function exposed by the package: