etlplus 0.8.3__tar.gz → 0.12.5__tar.gz

{etlplus-0.8.3 → etlplus-0.12.5}/DEMO.md

@@ -89,14 +89,14 @@ $ etlplus validate '{"email": "user@example.com", "age": 25}' \
 
 ### Filter and Select
 ```bash
-$ etlplus transform '[
+$ etlplus transform --operations '{
+    "filter": {"field": "age", "op": "gt", "value": 26},
+    "select": ["name", "age"]
+  }' '[
     {"name": "John", "age": 30, "city": "NYC"},
     {"name": "Jane", "age": 25, "city": "LA"},
     {"name": "Bob", "age": 35, "city": "Chicago"}
-  ]' --operations '{
-    "filter": {"field": "age", "op": "gt", "value": 26},
-    "select": ["name", "age"]
-  }'
+  ]'
 [
   {
     "name": "John",
@@ -111,24 +111,19 @@ $ etlplus transform '[
 
 ### Sort Data
 ```bash
-$ etlplus transform '[
-    {"name": "Charlie", "score": 85},
-    {"name": "Alice", "score": 95},
-    {"name": "Bob", "score": 90}
-  ]' --operations '{
-    "sort": {"field": "score", "reverse": true}
-  }'
+$ etlplus transform -\
+  -operations '{"sort": {"field": "score", "reverse": true}}' \
+  '[{"name": "Charlie", "score": 85}, {"name": "Alice", "score": 95}, {"name": "Bob", "score": 90}]'
 ```
 
 ### Aggregate Data
 ```bash
-$ etlplus transform '[
+$ etlplus transform --operations '{"aggregate": {"field": "sales", "func": "sum"}}' \
+  '[
     {"product": "A", "sales": 100},
     {"product": "B", "sales": 150},
     {"product": "C", "sales": 200}
-  ]' --operations '{
-    "aggregate": {"field": "sales", "func": "sum"}
-  }'
+  ]'
 {
   "sum_sales": 450
 }
@@ -138,7 +133,9 @@ $ etlplus transform '[
 
 ### Load to JSON File
 ```bash
-$ etlplus load '{"name": "John", "status": "active"}' file output.json
+$ etlplus load \
+  '{"name": "John", "status": "active"}' \
+  output.json --target-type file
 {
   "status": "success",
   "message": "Data loaded to output.json",
@@ -148,10 +145,12 @@ $ etlplus load '{"name": "John", "status": "active"}' file output.json
 
 ### Load to CSV File
 ```bash
-$ etlplus load '[
+$ etlplus load \
+  '[
     {"name": "John", "email": "john@example.com"},
     {"name": "Jane", "email": "jane@example.com"}
-  ]' --to users.csv
+  ]' \
+  users.csv --target-type file
 {
   "status": "success",
   "message": "Data loaded to users.csv",
@@ -173,19 +172,22 @@ This example shows a complete ETL workflow:
 $ etlplus extract raw_data.csv > extracted.json
 
 # Step 2: Transform
-$ etlplus transform --from extracted.json \
+$ etlplus transform \
   --operations '{
     "filter": {"field": "age", "op": "gte", "value": 18},
     "select": ["name", "email", "age"]
-  }' --to transformed.json
+  }' \
+  extracted.json \
+  transformed.json
 
 # Step 3: Validate
-$ etlplus validate transformed.json \
+$ etlplus validate \
   --rules '{
     "name": {"type": "string", "required": true},
     "email": {"type": "string", "required": true, "pattern": "^[\\w.-]+@[\\w.-]+\\.\\w+$"},
     "age": {"type": "number", "min": 18, "max": 120}
-  }'
+  }' \
+  transformed.json
 
 # Step 4: Load
 $ etlplus load transformed.json file final_output.csv

{etlplus-0.8.3/etlplus.egg-info → etlplus-0.12.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: etlplus
-Version: 0.8.3
+Version: 0.12.5
 Summary: A Swiss Army knife for simple ETL operations
 Home-page: https://github.com/Dagitali/ETLPlus
 Author: ETLPlus Team
@@ -17,8 +17,11 @@ Classifier: Programming Language :: Python :: 3.14
 Requires-Python: >=3.13,<3.15
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: fastavro>=1.12.1
 Requires-Dist: jinja2>=3.1.6
+Requires-Dist: openpyxl>=3.1.5
 Requires-Dist: pyodbc>=5.3.0
+Requires-Dist: pyarrow>=22.0.0
 Requires-Dist: python-dotenv>=1.2.1
 Requires-Dist: pandas>=2.3.3
 Requires-Dist: pydantic>=2.12.5
@@ -26,6 +29,8 @@ Requires-Dist: PyYAML>=6.0.3
 Requires-Dist: requests>=2.32.5
 Requires-Dist: SQLAlchemy>=2.0.45
 Requires-Dist: typer>=0.21.0
+Requires-Dist: xlrd>=2.0.2
+Requires-Dist: xlwt>=1.3.0
 Provides-Extra: dev
 Requires-Dist: black>=25.9.0; extra == "dev"
 Requires-Dist: build>=1.2.2; extra == "dev"
@@ -59,11 +64,13 @@ ETLPlus is a veritable Swiss Army knife for enabling simple ETL operations, offe
 package and command-line interface for data extraction, validation, transformation, and loading.
 
 - [ETLPlus](#etlplus)
+  - [Getting Started](#getting-started)
   - [Features](#features)
   - [Installation](#installation)
   - [Quickstart](#quickstart)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
+      - [Argument Order and Required Options](#argument-order-and-required-options)
       - [Check Pipelines](#check-pipelines)
       - [Render SQL DDL](#render-sql-ddl)
       - [Extract Data](#extract-data)
@@ -86,11 +93,27 @@ package and command-line interface for data extraction, validation, transformati
     - [Linting](#linting)
     - [Updating Demo Snippets](#updating-demo-snippets)
     - [Releasing to PyPI](#releasing-to-pypi)
-  - [Links](#links)
   - [License](#license)
   - [Contributing](#contributing)
+  - [Documentation](#documentation)
+    - [Python Packages/Subpackage](#python-packagessubpackage)
+    - [Community Health](#community-health)
+    - [Other](#other)
   - [Acknowledgments](#acknowledgments)
 
+## Getting Started
+
+ETLPlus helps you extract, validate, transform, and load data from files, databases, and APIs, either
+as a Python library or from the command line.
+
+To get started:
+
+- See [Installation](#installation) for setup instructions.
+- Try the [Quickstart](#quickstart) for a minimal working example (CLI and Python).
+- Explore [Usage](#usage) for more detailed options and workflows.
+
+ETLPlus supports Python 3.13 and above.
+
 ## Features
 
 - **Check** data pipeline definitions before running them:
@@ -151,8 +174,8 @@ etlplus --version
 
 # One-liner: extract CSV, filter, select, and write JSON
 etlplus extract file examples/data/sample.csv \
-  | etlplus transform - --operations '{"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}' \
-  -o temp/sample_output.json
+  | etlplus transform --operations '{"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}' \
+  - temp/sample_output.json
 ```
 
 [Python API](#python-api):
@@ -185,6 +208,27 @@ etlplus --version
 The CLI is implemented with Typer (Click-based). There is no argparse compatibility layer, so rely
 on the documented commands/flags and run `etlplus <command> --help` for current options.
 
+**Example error messages:**
+
+- If you omit a required argument: `Error: Missing required argument 'SOURCE'.`
+- If you place an option before its argument: `Error: Option '--source-format' must follow the 'SOURCE' argument.`
+
+#### Argument Order and Required Options
+
+For each command, positional arguments must precede options. Required options must follow their
+associated argument:
+
+- **extract**: `etlplus extract SOURCE [--source-format ...] [--source-type ...]`
+  - `SOURCE` is required. `--source-format` and `--source-type` must follow `SOURCE`.
+- **transform**: `etlplus transform [--operations ...] SOURCE [--source-format ...] [--source-type ...] TARGET [--target-format ...] [--target-type ...]`
+  - `SOURCE` and `TARGET` are required. Format/type options must follow their respective argument.
+- **load**: `etlplus load TARGET [--target-format ...] [--target-type ...] [--source-format ...]`
+  - `TARGET` is required. `--target-format` and `--target-type` must follow `TARGET`.
+- **validate**: `etlplus validate SOURCE [--rules ...] [--source-format ...] [--source-type ...]`
+  - `SOURCE` is required. `--rules` and format/type options must follow `SOURCE`.
+
+If required arguments or options are missing, or if options are placed before their associated argument, the CLI will display a clear error message.
+
 #### Check Pipelines
 
 Use `etlplus check` to explore pipeline YAML definitions without running them. The command can print
@@ -251,7 +295,7 @@ etlplus extract api https://api.example.com/data
 
 Save extracted data to file:
 ```bash
-etlplus extract file examples/data/sample.csv -o temp/sample_output.json
+etlplus extract file examples/data/sample.csv > temp/sample_output.json
 ```
 
 #### Validate Data
@@ -270,59 +314,67 @@ etlplus validate examples/data/sample.json --rules '{"email": {"type": "string",
 
 When piping data through `etlplus transform`, use `--source-format` whenever the SOURCE argument is
 `-` or a literal payload, mirroring the `etlplus extract` semantics. Use `--target-format` to
-control the emitted format for stdout or other non-file outputs, just like `etlplus load`. File
-paths continue to infer formats from their extensions. Use `--from` to override the inferred source
-connector type and `--to` to override the inferred target connector type, matching the `etlplus
-extract`/`etlplus load` behavior.
+control the emitted format for STDOUT or other non-file outputs, just like `etlplus load`. File
+paths continue to infer formats from their extensions. Use `--source-type` to override the inferred
+source connector type and `--target-type` to override the inferred target connector type, matching
+the `etlplus extract`/`etlplus load` behavior.
 
 Transform file inputs while overriding connector types:
 ```bash
-etlplus transform --from file examples/data/sample.json \
+etlplus transform \
   --operations '{"select": ["name", "email"]}' \
-  --to file -o temp/selected_output.json
+  examples/data/sample.json  --source-type file \
+  temp/selected_output.json --target-type file
 ```
 
 Filter and select fields:
 ```bash
-etlplus transform '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]' \
-  --operations '{"filter": {"field": "age", "op": "gt", "value": 26}, "select": ["name"]}'
+etlplus transform \
+  --operations '{"filter": {"field": "age", "op": "gt", "value": 26}, "select": ["name"]}' \
+  '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]'
 ```
 
 Sort data:
 ```bash
-etlplus transform examples/data/sample.json --operations '{"sort": {"field": "age", "reverse": true}}'
+etlplus transform \
+  --operations '{"sort": {"field": "age", "reverse": true}}' \
+  examples/data/sample.json
 ```
 
 Aggregate data:
 ```bash
-etlplus transform examples/data/sample.json --operations '{"aggregate": {"field": "age", "func": "sum"}}'
+etlplus transform \
+  --operations '{"aggregate": {"field": "age", "func": "sum"}}' \
+  examples/data/sample.json
 ```
 
 Map/rename fields:
 ```bash
-etlplus transform examples/data/sample.json --operations '{"map": {"name": "new_name"}}'
+etlplus transform \
+  --operations '{"map": {"name": "new_name"}}' \
+  examples/data/sample.json
 ```
 
 #### Load Data
 
-`etlplus load` consumes JSON from stdin; provide only the target argument plus optional flags.
+`etlplus load` consumes JSON from STDIN; provide only the target argument plus optional flags.
 
 Load to JSON file:
 ```bash
 etlplus extract file examples/data/sample.json \
-  | etlplus load --to file temp/sample_output.json
+  | etlplus load temp/sample_output.json --target-type file
 ```
 
 Load to CSV file:
 ```bash
 etlplus extract file examples/data/sample.csv \
-  | etlplus load --to file temp/sample_output.csv
+  | etlplus load temp/sample_output.csv --target-type file
 ```
 
 Load to REST API:
 ```bash
 cat examples/data/sample.json \
-  | etlplus load --to api https://api.example.com/endpoint
+  | etlplus load https://api.example.com/endpoint --target-type api
 ```
 
 ### Python API
@@ -375,20 +427,22 @@ etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers
 
 ```bash
 # 1. Extract from CSV
-etlplus extract file examples/data/sample.csv -o temp/sample_extracted.json
+etlplus extract file examples/data/sample.csv > temp/sample_extracted.json
 
 # 2. Transform (filter and select fields)
-etlplus transform temp/sample_extracted.json \
+etlplus transform \
   --operations '{"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}' \
-  -o temp/sample_transformed.json
+  temp/sample_extracted.json \
+  temp/sample_transformed.json
 
 # 3. Validate transformed data
-etlplus validate temp/sample_transformed.json \
-  --rules '{"name": {"type": "string", "required": true}, "email": {"type": "string", "required": true}}'
+etlplus validate \
+  --rules '{"name": {"type": "string", "required": true}, "email": {"type": "string", "required": true}}' \
+  temp/sample_transformed.json
 
 # 4. Load to CSV
 cat temp/sample_transformed.json \
-  | etlplus load --to temp/sample_output.csv
+  | etlplus load temp/sample_output.csv
 ```
 
 ### Format Overrides
@@ -401,14 +455,14 @@ Examples (zsh):
 
 ```zsh
 # Force CSV parsing for an extension-less file
-etlplus extract --from file data.txt --source-format csv
+etlplus extract data.txt --source-type file --source-format csv
 
 # Write CSV to a file without the .csv suffix
-etlplus load --to file output.bin --target-format csv < data.json
+etlplus load output.bin --target-type file --target-format csv < data.json
 
 # Leave the flags off when extensions already match the desired format
-etlplus extract --from file data.csv
-etlplus load --to file data.json < data.json
+etlplus extract data.csv --source-type file
+etlplus load data.json --target-type file  < data.json
 ```
 
 ## Transformation Operations
@@ -571,17 +625,6 @@ git push origin v1.4.0
 If you want an extra smoke-test before tagging, run `make dist && pip install dist/*.whl` locally;
 this exercises the same build path the workflow uses.
 
-## Links
-
-- API client docs: [`etlplus/api/README.md`](etlplus/api/README.md)
-- Examples: [`examples/README.md`](examples/README.md)
-- Pipeline authoring guide: [`docs/pipeline-guide.md`](docs/pipeline-guide.md)
-- Runner internals: [`docs/run-module.md`](docs/run-module.md)
-- Design notes (Mapping inputs, dict outputs): [`docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs`](docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs)
-- Typing philosophy: [`CONTRIBUTING.md#typing-philosophy`](CONTRIBUTING.md#typing-philosophy)
-- Demo and walkthrough: [`DEMO.md`](DEMO.md)
-- Additional references: [`REFERENCES.md`](`REFERENCES.md)
-
 ## License
 
 This project is licensed under the [MIT License](LICENSE).
@@ -605,6 +648,39 @@ If you choose to be a code contributor, please first refer these documents:
 - Typing philosophy (TypedDicts as editor hints, permissive runtime):
   [`CONTRIBUTING.md#typing-philosophy`](CONTRIBUTING.md#typing-philosophy)
 
+## Documentation
+
+### Python Packages/Subpackage
+
+Navigate to detailed documentation for each subpackage:
+
+- [etlplus.api](etlplus/api/README.md): Lightweight HTTP client and paginated REST helpers
+- [etlplus.file](etlplus/file/README.md): Unified file format support and helpers
+- [etlplus.config](etlplus/config/README.md): Configuration helpers for connectors, pipelines, jobs,
+  and profiles
+- [etlplus.cli](etlplus/cli/README.md): Command-line interface for ETLPlus workflows
+- [etlplus.database](etlplus/database/README.md): Database engine, schema, and ORM helpers
+- [etlplus.templates](etlplus/templates/README.md): SQL and DDL template helpers
+- [etlplus.validation](etlplus/validation/README.md): Data validation utilities and helpers
+
+### Community Health
+
+- [Contributing Guidelines](CONTRIBUTING.md): How to contribute, report issues, and submit PRs
+- [Code of Conduct](CODE_OF_CONDUCT.md): Community standards and expectations
+- [Security Policy](SECURITY.md): Responsible disclosure and vulnerability reporting
+- [Support](SUPPORT.md): Where to get help
+
+### Other
+
+- API client docs: [`etlplus/api/README.md`](etlplus/api/README.md)
+- Examples: [`examples/README.md`](examples/README.md)
+- Pipeline authoring guide: [`docs/pipeline-guide.md`](docs/pipeline-guide.md)
+- Runner internals: [`docs/run-module.md`](docs/run-module.md)
+- Design notes (Mapping inputs, dict outputs): [`docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs`](docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs)
+- Typing philosophy: [`CONTRIBUTING.md#typing-philosophy`](CONTRIBUTING.md#typing-philosophy)
+- Demo and walkthrough: [`DEMO.md`](DEMO.md)
+- Additional references: [`REFERENCES.md`](REFERENCES.md)
+
 ## Acknowledgments
 
 ETLPlus is inspired by common work patterns in data engineering and software engineering patterns in