easy-data-loader 0.1.4__tar.gz → 0.1.6__tar.gz

{easy_data_loader-0.1.4/src/easy_data_loader.egg-info → easy_data_loader-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: easy_data_loader
-Version: 0.1.4
+Version: 0.1.6
 Summary: Data transfer utilities between files and databases
 Author-email: Bojoi Gabriel <bojoigabriel@gmail.com>
 Classifier: Development Status :: 3 - Alpha
@@ -61,16 +61,26 @@ uv add easy_data_loader
    easy-data-loader init
    ```
 2. Review the generated `config/` folders for sample resources and pipelines.
-3. Run all discovered pipelines across the active configurations:
+3. List all discovered resources and pipelines:
    ```bash
-   easy-data-loader run_all
+   easy-data-loader list
    ```
 
 ## ✔️ Generic concepts
 
 `easy_data_loader` uses `resources` as a way to define a file or a database. The resouces can represent either a source or a destination making posible the folowing ETL scenarios: file -> file, file -> database, database -> file, database -> database.
 
-`easy_data_loader` project initializer will created the predefined folder structure `/config/resources` where the resources are expected to be defined following the current convention: the file type is .env and the file name must be prefixed with the resource type `file_` or `database_`. The predefined folder structure together with the naming convention enables `easy_data_loader` to find and load all resources.
+`easy_data_loader` project initializer will create the predefined folder structure `/config/resources`. Resource `.env` files are detected from their contents: database resources define `CONN_SERVER_TYPE`, while folder resources define `FOLDER_PATH`.
+
+Individual file definitions are regular Python objects in modules under `/config/resources`. They are declared as a flat dictionary, independent of any folder resource - the folder is only chosen when a pipeline references the file:
+
+```python
+FILES = {
+    "example_file": FileEntry(file_type=FileType.CSV, file_name="data"),
+}
+```
+
+A pipeline (or `resource_access` helper) references the file by combining a folder resource name with the file key: `"example_folder.example_file"`.
 
 A secondary predefined folder `/config/pipelines` will contain the pipeline definition files, which are regular Python files. There are 3 types of pipelines that can be defined:
 - `LoadPipeline` the main pipeline type which transports data from source to destination
@@ -80,8 +90,8 @@ A secondary predefined folder `/config/pipelines` will contain the pipeline defi
 ## LoadPipeline
 
 In order to define a `LoadPipeline` we must use the `BasePipelineDefinition` from `easy_data_loader` as depicted in the example pipelines created by the initializer.
+`pipeline_name` is never set by the user - it is derived automatically from the pipeline's file name at load time.
 In the simplest form there are only a few mandatory parameters:
-- `pipeline_name : str` - this name will be used to execute the pipeline
 - `source : str` - the file name (without extension) coresponding to the desired resource to be the data source
 - `destination : str` - the file name (without extension) coresponding to the desired resource to be the data destination
 
@@ -99,15 +109,13 @@ There are many other aspects of the pipeline that can be defined:
 ## ProcedurePipeline
 
 This secondary pipeline type is responsible for executing one or more stored procedures inside a database.
-To define one we need to use the `ProcedureDefinition` with the following parameters:
-- `pipeline_name : str` - this name will be used to execute the pipeline
+To define one we need to use the `ProcedureDefinition` with the following parameters (`pipeline_name` is derived automatically from the file name, same as `LoadPipeline`):
 - `audit : str, optional` - database resource name where the audit info will be recorded
 - `resource : str` - database resource name where the stored procedure(s) wil be executed
 - `procedures : List[tuple(str, Optional[Dict[str, Any]])]` - list of one or more stord procedures along with optional procedures parameters as dictionaries
 
 ## OrchestratorPipeline
 
-This pipeline type is responsible of executing sequentially a set of pipelines, `LoadPipeline`s and / or `ProcedurePipeline`s. Very simple to define using the `OrchestratorDefinition` with:
-- `orchestrator_name : str` - name by which the orchestrator is executer
-- 'pipelines : List[str]` - list of pipelines to execute sequentially
+This pipeline type is responsible of executing sequentially a set of pipelines, `LoadPipeline`s and / or `ProcedurePipeline`s. Very simple to define using the `OrchestratorDefinition` (`pipeline_name` is derived automatically from the file name) with:
+- `pipelines : List[str]` - list of pipelines to execute sequentially
 - `fail_fast : bool, Default True` - if any of the pipelines fail the rest of the pipelines in the list do not get executed

{easy_data_loader-0.1.4 → easy_data_loader-0.1.6}/README.md

@@ -32,16 +32,26 @@ uv add easy_data_loader
    easy-data-loader init
    ```
 2. Review the generated `config/` folders for sample resources and pipelines.
-3. Run all discovered pipelines across the active configurations:
+3. List all discovered resources and pipelines:
    ```bash
-   easy-data-loader run_all
+   easy-data-loader list
    ```
 
 ## ✔️ Generic concepts
 
 `easy_data_loader` uses `resources` as a way to define a file or a database. The resouces can represent either a source or a destination making posible the folowing ETL scenarios: file -> file, file -> database, database -> file, database -> database.
 
-`easy_data_loader` project initializer will created the predefined folder structure `/config/resources` where the resources are expected to be defined following the current convention: the file type is .env and the file name must be prefixed with the resource type `file_` or `database_`. The predefined folder structure together with the naming convention enables `easy_data_loader` to find and load all resources.
+`easy_data_loader` project initializer will create the predefined folder structure `/config/resources`. Resource `.env` files are detected from their contents: database resources define `CONN_SERVER_TYPE`, while folder resources define `FOLDER_PATH`.
+
+Individual file definitions are regular Python objects in modules under `/config/resources`. They are declared as a flat dictionary, independent of any folder resource - the folder is only chosen when a pipeline references the file:
+
+```python
+FILES = {
+    "example_file": FileEntry(file_type=FileType.CSV, file_name="data"),
+}
+```
+
+A pipeline (or `resource_access` helper) references the file by combining a folder resource name with the file key: `"example_folder.example_file"`.
 
 A secondary predefined folder `/config/pipelines` will contain the pipeline definition files, which are regular Python files. There are 3 types of pipelines that can be defined:
 - `LoadPipeline` the main pipeline type which transports data from source to destination
@@ -51,8 +61,8 @@ A secondary predefined folder `/config/pipelines` will contain the pipeline defi
 ## LoadPipeline
 
 In order to define a `LoadPipeline` we must use the `BasePipelineDefinition` from `easy_data_loader` as depicted in the example pipelines created by the initializer.
+`pipeline_name` is never set by the user - it is derived automatically from the pipeline's file name at load time.
 In the simplest form there are only a few mandatory parameters:
-- `pipeline_name : str` - this name will be used to execute the pipeline
 - `source : str` - the file name (without extension) coresponding to the desired resource to be the data source
 - `destination : str` - the file name (without extension) coresponding to the desired resource to be the data destination
 
@@ -70,15 +80,13 @@ There are many other aspects of the pipeline that can be defined:
 ## ProcedurePipeline
 
 This secondary pipeline type is responsible for executing one or more stored procedures inside a database.
-To define one we need to use the `ProcedureDefinition` with the following parameters:
-- `pipeline_name : str` - this name will be used to execute the pipeline
+To define one we need to use the `ProcedureDefinition` with the following parameters (`pipeline_name` is derived automatically from the file name, same as `LoadPipeline`):
 - `audit : str, optional` - database resource name where the audit info will be recorded
 - `resource : str` - database resource name where the stored procedure(s) wil be executed
 - `procedures : List[tuple(str, Optional[Dict[str, Any]])]` - list of one or more stord procedures along with optional procedures parameters as dictionaries
 
 ## OrchestratorPipeline
 
-This pipeline type is responsible of executing sequentially a set of pipelines, `LoadPipeline`s and / or `ProcedurePipeline`s. Very simple to define using the `OrchestratorDefinition` with:
-- `orchestrator_name : str` - name by which the orchestrator is executer
-- 'pipelines : List[str]` - list of pipelines to execute sequentially
+This pipeline type is responsible of executing sequentially a set of pipelines, `LoadPipeline`s and / or `ProcedurePipeline`s. Very simple to define using the `OrchestratorDefinition` (`pipeline_name` is derived automatically from the file name) with:
+- `pipelines : List[str]` - list of pipelines to execute sequentially
 - `fail_fast : bool, Default True` - if any of the pipelines fail the rest of the pipelines in the list do not get executed

{easy_data_loader-0.1.4 → easy_data_loader-0.1.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "easy_data_loader"
-version = "0.1.4"
+version = "0.1.6"
 description = "Data transfer utilities between files and databases"
 authors = [{ name = "Bojoi Gabriel", email = "bojoigabriel@gmail.com" }]
 readme = "README.md"

{easy_data_loader-0.1.4 → easy_data_loader-0.1.6}/src/easy_data_loader/cli.py

@@ -1,28 +1,33 @@
 from pathlib import Path
 
 import click
+from .utils import to_snake_case
 
 # Integrated templates
 PIPELINE_TEMPLATE = """
-from easy_data_loader.pipeline import LoadPipeline
 from easy_data_loader.models import BasePipelineDefinition, ColumnDefinition
 import pandas as pd
-from sqlalchemy.types import DATETIME, INT, DECIMAL, NVARCHAR
 
 example_pipeline = BasePipelineDefinition(
-    pipeline_name="test_pipeline", # pipeline name to be used when initializing the Pipeline object
 
-    # source name represented by the file having the source settings -> corresponds to a file in the config/resources folder
-    source = "example_file",
-
-    # "dbo.FactSales" ## if the source is a database then source sql defines the table name or a custom query to be executed
+<<<<<<< HEAD
+    # if the source is a database then fill with the file name (witout extension) of the resource definition in config/resources (e.g. example_database)
+    # if the source is a file then the source name is represented by "<folder_resource_name>.<file_key>" -> the folder resource
+=======
+    # source name represented by "<folder_resource_name>.<file_key>" -> the folder resource
+>>>>>>> main
+    # corresponds to a file in config/resources (e.g. example_folder.env) and the file_key
+    # is defined in config/resources/files.py
+    source = "example_folder.example_file",
+
+    # [optional] only if the source is a database then here we define the table name for a 1:1 copy or a custom sql query
     # source_sql = "SELECT TOP 100 * FROM dbo.FactSales"
 
-    # destination name represented by the file having the destination settings -> corresponds to a file in the config/resources folder
+    # fill with the same logic as source
     destination="example_database",
 
     # if the destination is a database then here we define the destination table name
-    destination_table="dbo.LargeSalesData",
+    destination_table="example_table",
 
     # columns definition if we are sending data to a database table
     columns={
@@ -35,13 +40,16 @@ example_pipeline = BasePipelineDefinition(
         "raw_notes" : ColumnDefinition(target_name="notes", data_type=NVARCHAR(100))
     },
 
-    # different parameters passed to the write functions
+    # different parameters passed to the write functions to be forwarded to the pandas function that will write to destination (e.g. to_sql, to_csv, to_excel, etc.)
     write_parameters={"if_exists" : "replace", "index" : False},
 
-    # different parameters passed to the read function
+    # different parameters passed to the read function to be forwarded to the pandas function that will read from source (e.g. read_csv, read_excel, read_sql etc.)
     read_parameters={"sep" : ";"}
 )
 
+# we can define custom transformations that will be applied to the DataFrame before writing to the destination. 
+# the function must take a DataFrame as input and return a DataFrame as output.
+
 def add_timestamp(df):
     # Adding an audit column during load
     df['insert_timestamp'] = pd.Timestamp.now()
@@ -54,7 +62,7 @@ PROCEDURE_TEMPLATE = """
 from easy_data_loader.models import ProcedureDefinition
 
 example_procedure = ProcedureDefinition(
-    pipeline_name="example_procedure",
+
     resource="example_database",
     procedures=[
         ("dbo.sp_UpdateSales", {"year": 2024}),
@@ -67,7 +75,6 @@ ORCHESTRATOR_TEMPLATE = """
 from easy_data_loader.models import OrchestratorDefinition
 
 example_orchestrator = OrchestratorDefinition(
-    orchestrator_name="example_orchestrator",
     pipelines=[
         "example_pipeline",
         "example_procedure"
@@ -78,7 +85,8 @@ example_orchestrator = OrchestratorDefinition(
 
 
 DATABASE_ENV = """
-# database resource definition
+# database resource definition (type is auto-detected from CONN_SERVER_TYPE)
+# database resource definition (type is auto-detected from CONN_SERVER_TYPE)
 CONN_SERVER_TYPE=MSSQL
 CONN_SERVER=.
 CONN_DATABASE=test_database
@@ -87,23 +95,48 @@ CONN_PASSWORD=my_password
 CONN_PORT=1433
 """
 
-FILE_ENV = """
-# file resource definition
-FILE_TYPE=CSV                # can also be XLSX, PARQUET, ORC
-FOLDER_PATH=./data/imports   # source folder where the file is located
-FILE_NAME=large_sales_data   # exact file name without extension
-#FILE_PATTERN=large_sales    # file pattern to search in the source folder
+FOLDER_ENV = """
+# folder resource definition
+# environment-specific can be absolute path or a relative path to the project root
+FOLDER_PATH=./data/imports
+"""
+
+FILES_TEMPLATE = """
+from easy_data_loader.models import FileEntry, FileType
+
+# File definitions are flat and independent of any folder resource. A folder is
+# only chosen when a pipeline references the file as
+# "<folder_resource_name>.<key>", e.g. "example_folder.example_file"
+
+FILES = {
+    "example_file": FileEntry(
+        # supported values: FileType.CSV, FileType.XLSX, FileType.ORC, FileType.PARQUET
+        file_type=FileType.CSV,
+
+        # if the file name never changes we can use an exact name without the extension
+        file_name="large_sales_data",
+
+        # if the file name changes we can use a pattern in glob format; if multiple files matching the pattern, the most recent one will be used
+        # file_pattern="*large_sales*",
+    ),
+}
 """
 
 MAIN = """
-from easy_data_loader import LoadPipeline, ProcedurePipeline
+from easy_data_loader import LoadPipeline, ProcedurePipeline, OrchestratorPipeline
+
+# example execution of different types of pipelines using the easy data loader library
 
 def main():
     # Run an ETL pipeline
-    LoadPipeline(pipeline_name="example_pipeline").run()
+    first_step = LoadPipeline(pipeline_name="example_pipeline").run()
 
     # Run a procedure pipeline
-    ProcedurePipeline(pipeline_name="example_procedure").run()
+    if first_step:
+        ProcedurePipeline(pipeline_name="example_procedure").run()
+
+    # Run an orchestrator pipeline
+    OrchestratorPipeline(pipeline_name="example_orchestrator").run()
 
 if __name__ == "__main__":
     main()
@@ -128,12 +161,13 @@ def init():
 
     # Example files to create
     files = {
-        "config/pipelines/pipeline_example.py": PIPELINE_TEMPLATE,
-        "config/pipelines/procedure_example.py": PROCEDURE_TEMPLATE,
-        "config/pipelines/orchestrator_example.py": ORCHESTRATOR_TEMPLATE,
-        "config/resources/database_example.env": DATABASE_ENV,
-        "config/resources/file_example.env": FILE_ENV,
-        "main.py": MAIN,
+        "config/pipelines/pipeline.py.example": PIPELINE_TEMPLATE,
+        "config/pipelines/procedure.py.example": PROCEDURE_TEMPLATE,
+        "config/pipelines/orchestrator.py.example": ORCHESTRATOR_TEMPLATE,
+        "config/resources/database.env.example": DATABASE_ENV,
+        "config/resources/folder.env.example": FOLDER_ENV,
+        "config/resources/files.py": FILES_TEMPLATE,
+        "pipelines_start.py": MAIN,
     }
 
     for name, content in files.items():
@@ -159,6 +193,12 @@ def list():
     for name in config.get_all_resources():
         click.echo(f" - {name}")
 
+    click.echo(
+        "\n--- Discovered File Entries (combine with any folder as <folder>.<key>) ---"
+    )
+    for name in config.get_all_file_entries():
+        click.echo(f" - {name}")
+
     click.echo("\n--- Discovered Pipelines ---")
     for name in config.get_all_pipelines():
         click.echo(f" - {name}")
@@ -167,7 +207,7 @@ def list():
 @main.command()
 @click.argument("resource_name")
 @click.argument("table_name")
-def inspect_db(resource_name, table_name):
+def inspect_database_table(resource_name, table_name):
     """Inspect a database table and generate ColumnDefinition code"""
     from .config_loader import Configuration
     from .database_connector import CONNECTOR_FACTORY
@@ -204,70 +244,38 @@ def inspect_db(resource_name, table_name):
 
 
 @main.command()
-def run_all():
-    """Run all discovered pipelines and show status summary"""
+@click.argument("resource_name")
+def inspect_file_resource(resource_name):
+    """Inspect a file resource and generate ColumnDefinition code"""
     from .config_loader import Configuration
-    from .models import (
-        BasePipelineDefinition,
-        OrchestratorDefinition,
-        ProcedureDefinition,
-    )
-    from .pipeline import LoadPipeline
-    from .procedure_pipeline import ProcedurePipeline
+    from .file_operations import FileOperations
+    from .models import FileSettings, FileType
 
     config = Configuration()
-    pipelines = config.get_all_pipelines()
+    resource = config.get_resource(resource_name)
 
-    if not pipelines:
-        click.echo("No pipelines discovered.")
+    if not isinstance(resource, FileSettings):
+        click.echo(f"Error: Resource '{resource_name}' is not a file resource.")
         return
 
-    results = {}
-
-    click.echo(f"🚀 Running {len(pipelines)} discovered pipelines...\n")
-
-    for name in pipelines:
-        click.echo(f"Pipeline: {name} ... ", nl=False)
-        try:
-            definition = config.get_pipeline(name)
-            if isinstance(definition, BasePipelineDefinition):
-                success = LoadPipeline(name).run()
-            elif isinstance(definition, ProcedureDefinition):
-                success = ProcedurePipeline(name).run()
-            elif isinstance(definition, OrchestratorDefinition):
-                from .orchestrator import OrchestratorPipeline
-
-                success = OrchestratorPipeline(name).run()
-            else:
-                success = False
-
-            results[name] = "SUCCESS" if success else "FAILED"
-        except Exception as e:
-            results[name] = f"ERROR: {str(e)}"
-
-        click.echo(results[name])
-
-    click.echo("\n" + "=" * 40)
-    click.echo(f"{'PIPELINE':<25} | {'STATUS'}")
-    click.echo("-" * 40)
-    for name, status in results.items():
-        click.echo(f"{name:<25} | {status}")
+    ops = FileOperations(resource)
+    df = ops.read_file()
 
+    schema = ops._inferencers[ops.file_type](
+        df if ops.file_type in [FileType.CSV, FileType.XLSX] else ops.file_path
+    )
 
-@main.command()
-@click.argument("orchestrator_name")
-def run_orchestrator(orchestrator_name):
-    """Run a specific orchestrator by name"""
-    from .orchestrator import OrchestratorPipeline
+    if not schema:
+        click.echo(f"No columns found for resource '{resource_name}'.")
+        return
 
-    try:
-        success = OrchestratorPipeline(orchestrator_name).run()
-        if success:
-            click.echo(f"✅ Orchestrator '{orchestrator_name}' completed successfully.")
-        else:
-            click.echo(f"❌ Orchestrator '{orchestrator_name}' failed.")
-    except Exception as e:
-        click.echo(f"💥 Error: {str(e)}")
+    click.echo(f"\n# Suggested Column definitions for {resource_name}:")
+    click.echo("columns={")
+    for col, dtype in schema.items():
+        click.echo(
+            f'    "{col}": ColumnDefinition(target_name="{to_snake_case(col)}", data_type={dtype}),'
+        )
+    click.echo("}")
 
 
 @main.command()
@@ -275,7 +283,7 @@ def validate_resources():
     """Validate all configured resources"""
     from .config_loader import Configuration
     from .database_connector import CONNECTOR_FACTORY
-    from .models import FileSettings
+    from .models import FileSettings, FolderSettings
 
     config = Configuration()
     resources = config.get_all_resources()
@@ -306,6 +314,12 @@ def validate_resources():
                     results[name] = "OK (Path Exists)"
                 else:
                     raise ValueError(f"Path does not exist: {resource.folder_path}")
+            elif isinstance(resource, FolderSettings):
+                # Validate Folder Path
+                if resource.folder_path.exists():
+                    results[name] = "OK (Path Exists)"
+                else:
+                    raise ValueError(f"Path does not exist: {resource.folder_path}")
             else:
                 results[name] = "UNKNOWN TYPE"
 
@@ -321,5 +335,64 @@ def validate_resources():
         click.echo(f"{name:<30} | {status}")
 
 
+@main.command()
+def validate_pipelines():
+    """Validate pipeline instantiation and their connected resources (including folder.file_key combinations)"""
+    from typing import Optional, Union
+
+    from .config_loader import Configuration
+    from .models import (
+        BasePipelineDefinition,
+        OrchestratorDefinition,
+        ProcedureDefinition,
+    )
+    from .orchestrator import OrchestratorPipeline
+    from .pipeline import LoadPipeline
+    from .procedure_pipeline import ProcedurePipeline
+
+    config = Configuration()
+    pipelines = config.get_all_pipelines()
+
+    if not pipelines:
+        click.echo("No pipelines found.")
+        return
+
+    click.echo(f"🔍 Validating {len(pipelines)} pipelines...\n")
+
+    results = {}
+
+    for name, definition in pipelines.items():
+        click.echo(f"Pipeline: {name} ... ", nl=False)
+        instance: Optional[Union[LoadPipeline, ProcedurePipeline]] = None
+        try:
+            if isinstance(definition, BasePipelineDefinition):
+                instance = LoadPipeline(name)
+            elif isinstance(definition, ProcedureDefinition):
+                instance = ProcedurePipeline(name)
+            elif isinstance(definition, OrchestratorDefinition):
+                OrchestratorPipeline(name)
+                # instantiation alone does not touch nested pipelines, so validate them too
+                for nested_name in definition.pipelines:
+                    config.get_pipeline(nested_name)
+            else:
+                raise ValueError(f"Unknown pipeline type: {type(definition)}")
+
+            results[name] = "OK"
+        except Exception as e:
+            results[name] = f"FAILED: {str(e)}"
+        finally:
+            # dispose any database connectors opened during instantiation
+            if instance is not None:
+                instance._cleanup()
+
+        click.echo(results[name])
+
+    click.echo("\n" + "=" * 60)
+    click.echo(f"{'PIPELINE':<30} | {'STATUS'}")
+    click.echo("-" * 60)
+    for name, status in results.items():
+        click.echo(f"{name:<30} | {status}")
+
+
 if __name__ == "__main__":
     main()