informatica-python 1.4.1__tar.gz → 1.5.0__tar.gz

informatica_python-1.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Nick
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

informatica_python-1.5.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: informatica-python
+Version: 1.5.0
+Summary: Convert Informatica PowerCenter workflow XML to Python/PySpark code
+Author: Nick
+License: MIT
+Keywords: informatica,powercenter,etl,code-generator,pandas,pyspark,data-engineering
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Code Generators
+Classifier: Topic :: Database :: Database Engines/Servers
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: lxml>=4.9.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Dynamic: license-file
+
+# informatica-python
+
+Convert Informatica PowerCenter workflow XML exports into clean, runnable Python/PySpark code.
+
+**Author:** Nick
+**License:** MIT
+**PyPI:** [informatica-python](https://pypi.org/project/informatica-python/)
+
+---
+
+## Overview
+
+`informatica-python` parses Informatica PowerCenter XML export files and generates equivalent Python code using your choice of data library. It handles all 72 DTD tags from the PowerCenter XML schema and produces a complete, ready-to-run Python project.
+
+## Installation
+
+```bash
+pip install informatica-python
+```
+
+## Quick Start
+
+### Command Line
+
+```bash
+# Generate Python files to a directory
+informatica-python workflow_export.xml -o output_dir
+
+# Generate as a zip archive
+informatica-python workflow_export.xml -z output.zip
+
+# Use a different data library
+informatica-python workflow_export.xml -o output_dir --data-lib polars
+
+# Parse to JSON only (no code generation)
+informatica-python workflow_export.xml --json
+
+# Save parsed JSON to file
+informatica-python workflow_export.xml --json-file parsed.json
+```
+
+### Python API
+
+```python
+from informatica_python import InformaticaConverter
+
+converter = InformaticaConverter()
+
+# Parse and generate files
+converter.convert_to_files("workflow_export.xml", "output_dir")
+
+# Parse and generate zip
+converter.convert_to_zip("workflow_export.xml", "output.zip")
+
+# Parse to structured dict
+result = converter.parse_file("workflow_export.xml")
+
+# Use a different data library
+converter.convert_to_files("workflow_export.xml", "output_dir", data_lib="polars")
+```
+
+## Generated Output Files
+
+| File | Description |
+|------|-------------|
+| `helper_functions.py` | Database/file I/O helpers, Informatica expression equivalents (80+ functions) |
+| `mapping_N.py` | One per mapping — transformation logic, source reads, target writes |
+| `workflow.py` | Task orchestration with topological ordering and error handling |
+| `config.yml` | Connection configs, source/target metadata, runtime parameters |
+| `all_sql_queries.sql` | All SQL extracted from Source Qualifiers, Lookups, SQL transforms |
+| `error_log.txt` | Conversion summary, warnings, and unsupported feature notes |
+
+## Supported Data Libraries
+
+Select via `--data-lib` CLI flag or `data_lib` parameter:
+
+| Library | Flag | Best For |
+|---------|------|----------|
+| **pandas** | `pandas` (default) | General-purpose, most compatible |
+| **dask** | `dask` | Large datasets, parallel processing |
+| **polars** | `polars` | High performance, Rust-backed |
+| **vaex** | `vaex` | Out-of-core, billion-row datasets |
+| **modin** | `modin` | Drop-in pandas replacement, multi-core |
+
+## Supported Transformations
+
+The code generator produces real, runnable Python for these transformation types:
+
+- **Source Qualifier** — SQL override, pre/post SQL, column selection
+- **Expression** — Field-level expressions converted to pandas operations
+- **Filter** — Row filtering with converted conditions
+- **Joiner** — `pd.merge()` with join type and condition parsing
+- **Lookup** — `pd.merge()` lookups with connection-aware DB/file reads
+- **Aggregator** — `groupby().agg()` with SUM/COUNT/AVG/MIN/MAX/FIRST/LAST
+- **Sorter** — `sort_values()` with multi-key ascending/descending
+- **Router** — Multi-group conditional routing with if/elif/else
+- **Union** — `pd.concat()` across multiple input groups
+- **Update Strategy** — Insert/Update/Delete/Reject flag generation
+- **Sequence Generator** — Auto-incrementing ID columns
+- **Normalizer** — `pd.melt()` with auto-detected id/value vars
+- **Rank** — `groupby().rank()` with Top-N filtering
+- **Stored Procedure** — Stub generation with SP name and parameters
+- **Transaction Control** — Commit/rollback logic stubs
+- **Custom / Java** — Placeholder stubs with TODO markers
+- **SQL Transform** — Direct SQL execution pass-through
+
+## Supported XML Tags (72 Tags)
+
+**Top-level:** POWERMART, REPOSITORY, FOLDER, FOLDERVERSION
+
+**Source/Target:** SOURCE, SOURCEFIELD, TARGET, TARGETFIELD, TARGETINDEX, TARGETINDEXFIELD, FLATFILE, XMLINFO, XMLTEXT, GROUP, TABLEATTRIBUTE, FIELDATTRIBUTE, METADATAEXTENSION, KEYWORD, ERPSRCINFO
+
+**Mapping/Mapplet:** MAPPING, MAPPLET, TRANSFORMATION, TRANSFORMFIELD, TRANSFORMFIELDATTR, TRANSFORMFIELDATTRDEF, INSTANCE, ASSOCIATED_SOURCE_INSTANCE, CONNECTOR, MAPDEPENDENCY, TARGETLOADORDER, MAPPINGVARIABLE, FIELDDEPENDENCY, INITPROP, ERPINFO
+
+**Task/Session/Workflow:** TASK, TIMER, VALUEPAIR, SCHEDULER, SCHEDULEINFO, STARTOPTIONS, ENDOPTIONS, SCHEDULEOPTIONS, RECURRING, CUSTOM, DAILYFREQUENCY, REPEAT, FILTER, SESSION, CONFIGREFERENCE, SESSTRANSFORMATIONINST, SESSTRANSFORMATIONGROUP, PARTITION, HASHKEY, KEYRANGE, CONFIG, SESSIONCOMPONENT, CONNECTIONREFERENCE, TASKINSTANCE, WORKFLOWLINK, WORKFLOWVARIABLE, WORKFLOWEVENT, WORKLET, WORKFLOW, ATTRIBUTE
+
+**Shortcut:** SHORTCUT
+
+**SAP:** SAPFUNCTION, SAPSTRUCTURE, SAPPROGRAM, SAPOUTPUTPORT, SAPVARIABLE, SAPPROGRAMFLOWOBJECT, SAPTABLEPARAM
+
+## Key Features
+
+### Session Connection Overrides (v1.4+)
+When sessions define per-transform connection overrides (different database, file directory, or filename), the generated code uses those overrides instead of source/target defaults.
+
+### Worklet Support (v1.4+)
+Worklet workflows are detected and generate separate `run_worklet_NAME(config)` functions. The main workflow calls these automatically for Worklet task types.
+
+### Type Casting at Target Writes (v1.4+)
+Target field datatypes are mapped to pandas types and generate proper casting code:
+- Integers: nullable `Int64`/`Int32` or `fillna(0).astype(int)` for NOT NULL
+- Dates: `pd.to_datetime(errors='coerce')`
+- Decimals/Floats: `pd.to_numeric(errors='coerce')`
+- Booleans: `.astype('boolean')`
+
+### Flat File Handling (v1.3+)
+Parses FLATFILE metadata for delimiter, fixed-width, header lines, skip rows, quote/escape chars. Generates `pd.read_fwf()` for fixed-width or enriched `read_file()` for delimited.
+
+### Mapplet Inlining (v1.3+)
+Expands Mapplet instances into prefixed transforms, rewires connectors, and eliminates duplication.
+
+### Decision Tasks (v1.3+)
+Converts Informatica decision conditions to Python if/else branches with proper variable substitution.
+
+### Expression Converter (80+ Functions)
+
+Converts Informatica expressions to Python equivalents:
+
+- **String:** SUBSTR, LTRIM, RTRIM, UPPER, LOWER, LPAD, RPAD, INSTR, LENGTH, CONCAT, REPLACE, REG_EXTRACT, REG_REPLACE, REVERSE, INITCAP, CHR, ASCII
+- **Date:** ADD_TO_DATE, DATE_DIFF, GET_DATE_PART, SYSDATE, SYSTIMESTAMP, TO_DATE, TO_CHAR, TRUNC (date)
+- **Numeric:** ROUND, TRUNC, MOD, ABS, CEIL, FLOOR, POWER, SQRT, LOG, EXP, SIGN
+- **Conversion:** TO_INTEGER, TO_BIGINT, TO_FLOAT, TO_DECIMAL, TO_CHAR, TO_DATE
+- **Null handling:** IIF, DECODE, NVL, NVL2, ISNULL, IS_SPACES, IS_NUMBER
+- **Aggregate:** SUM, AVG, COUNT, MIN, MAX, FIRST, LAST, MEDIAN, STDDEV, VARIANCE
+- **Lookup:** :LKP expressions with dynamic lookup references
+- **Variable:** SETVARIABLE / mapping variable assignment
+
+## Requirements
+
+- Python >= 3.8
+- lxml >= 4.9.0
+- PyYAML >= 6.0
+
+## Changelog
+
+### v1.4.x (Phase 3)
+- Session connection overrides for sources and targets
+- Worklet function generation with safe invocation
+- Type casting at target writes based on TARGETFIELD datatypes
+- Flat-file session path overrides properly wired
+
+### v1.3.x (Phase 2)
+- FLATFILE metadata in source reads and target writes
+- Normalizer with `pd.melt()`
+- Rank with group-by and Top-N filtering
+- Decision tasks with real if/else branches
+- Mapplet instance inlining
+
+### v1.2.x (Phase 1)
+- Core parser for all 72 XML tags
+- Expression converter with 80+ functions
+- Aggregator, Joiner, Lookup code generation
+- Workflow orchestration with topological task ordering
+- Multi-library support (pandas, dask, polars, vaex, modin)
+
+## Development
+
+```bash
+# Clone and install in development mode
+cd informatica_python
+pip install -e ".[dev]"
+
+# Run tests (25 tests)
+pytest tests/test_converter.py -v
+```
+
+## License
+
+MIT License - Copyright (c) 2025 Nick
+
+See [LICENSE](LICENSE) for details.

informatica_python-1.5.0/README.md

@@ -0,0 +1,201 @@
+# informatica-python
+
+Convert Informatica PowerCenter workflow XML exports into clean, runnable Python/PySpark code.
+
+**Author:** Nick
+**License:** MIT
+**PyPI:** [informatica-python](https://pypi.org/project/informatica-python/)
+
+---
+
+## Overview
+
+`informatica-python` parses Informatica PowerCenter XML export files and generates equivalent Python code using your choice of data library. It handles all 72 DTD tags from the PowerCenter XML schema and produces a complete, ready-to-run Python project.
+
+## Installation
+
+```bash
+pip install informatica-python
+```
+
+## Quick Start
+
+### Command Line
+
+```bash
+# Generate Python files to a directory
+informatica-python workflow_export.xml -o output_dir
+
+# Generate as a zip archive
+informatica-python workflow_export.xml -z output.zip
+
+# Use a different data library
+informatica-python workflow_export.xml -o output_dir --data-lib polars
+
+# Parse to JSON only (no code generation)
+informatica-python workflow_export.xml --json
+
+# Save parsed JSON to file
+informatica-python workflow_export.xml --json-file parsed.json
+```
+
+### Python API
+
+```python
+from informatica_python import InformaticaConverter
+
+converter = InformaticaConverter()
+
+# Parse and generate files
+converter.convert_to_files("workflow_export.xml", "output_dir")
+
+# Parse and generate zip
+converter.convert_to_zip("workflow_export.xml", "output.zip")
+
+# Parse to structured dict
+result = converter.parse_file("workflow_export.xml")
+
+# Use a different data library
+converter.convert_to_files("workflow_export.xml", "output_dir", data_lib="polars")
+```
+
+## Generated Output Files
+
+| File | Description |
+|------|-------------|
+| `helper_functions.py` | Database/file I/O helpers, Informatica expression equivalents (80+ functions) |
+| `mapping_N.py` | One per mapping — transformation logic, source reads, target writes |
+| `workflow.py` | Task orchestration with topological ordering and error handling |
+| `config.yml` | Connection configs, source/target metadata, runtime parameters |
+| `all_sql_queries.sql` | All SQL extracted from Source Qualifiers, Lookups, SQL transforms |
+| `error_log.txt` | Conversion summary, warnings, and unsupported feature notes |
+
+## Supported Data Libraries
+
+Select via `--data-lib` CLI flag or `data_lib` parameter:
+
+| Library | Flag | Best For |
+|---------|------|----------|
+| **pandas** | `pandas` (default) | General-purpose, most compatible |
+| **dask** | `dask` | Large datasets, parallel processing |
+| **polars** | `polars` | High performance, Rust-backed |
+| **vaex** | `vaex` | Out-of-core, billion-row datasets |
+| **modin** | `modin` | Drop-in pandas replacement, multi-core |
+
+## Supported Transformations
+
+The code generator produces real, runnable Python for these transformation types:
+
+- **Source Qualifier** — SQL override, pre/post SQL, column selection
+- **Expression** — Field-level expressions converted to pandas operations
+- **Filter** — Row filtering with converted conditions
+- **Joiner** — `pd.merge()` with join type and condition parsing
+- **Lookup** — `pd.merge()` lookups with connection-aware DB/file reads
+- **Aggregator** — `groupby().agg()` with SUM/COUNT/AVG/MIN/MAX/FIRST/LAST
+- **Sorter** — `sort_values()` with multi-key ascending/descending
+- **Router** — Multi-group conditional routing with if/elif/else
+- **Union** — `pd.concat()` across multiple input groups
+- **Update Strategy** — Insert/Update/Delete/Reject flag generation
+- **Sequence Generator** — Auto-incrementing ID columns
+- **Normalizer** — `pd.melt()` with auto-detected id/value vars
+- **Rank** — `groupby().rank()` with Top-N filtering
+- **Stored Procedure** — Stub generation with SP name and parameters
+- **Transaction Control** — Commit/rollback logic stubs
+- **Custom / Java** — Placeholder stubs with TODO markers
+- **SQL Transform** — Direct SQL execution pass-through
+
+## Supported XML Tags (72 Tags)
+
+**Top-level:** POWERMART, REPOSITORY, FOLDER, FOLDERVERSION
+
+**Source/Target:** SOURCE, SOURCEFIELD, TARGET, TARGETFIELD, TARGETINDEX, TARGETINDEXFIELD, FLATFILE, XMLINFO, XMLTEXT, GROUP, TABLEATTRIBUTE, FIELDATTRIBUTE, METADATAEXTENSION, KEYWORD, ERPSRCINFO
+
+**Mapping/Mapplet:** MAPPING, MAPPLET, TRANSFORMATION, TRANSFORMFIELD, TRANSFORMFIELDATTR, TRANSFORMFIELDATTRDEF, INSTANCE, ASSOCIATED_SOURCE_INSTANCE, CONNECTOR, MAPDEPENDENCY, TARGETLOADORDER, MAPPINGVARIABLE, FIELDDEPENDENCY, INITPROP, ERPINFO
+
+**Task/Session/Workflow:** TASK, TIMER, VALUEPAIR, SCHEDULER, SCHEDULEINFO, STARTOPTIONS, ENDOPTIONS, SCHEDULEOPTIONS, RECURRING, CUSTOM, DAILYFREQUENCY, REPEAT, FILTER, SESSION, CONFIGREFERENCE, SESSTRANSFORMATIONINST, SESSTRANSFORMATIONGROUP, PARTITION, HASHKEY, KEYRANGE, CONFIG, SESSIONCOMPONENT, CONNECTIONREFERENCE, TASKINSTANCE, WORKFLOWLINK, WORKFLOWVARIABLE, WORKFLOWEVENT, WORKLET, WORKFLOW, ATTRIBUTE
+
+**Shortcut:** SHORTCUT
+
+**SAP:** SAPFUNCTION, SAPSTRUCTURE, SAPPROGRAM, SAPOUTPUTPORT, SAPVARIABLE, SAPPROGRAMFLOWOBJECT, SAPTABLEPARAM
+
+## Key Features
+
+### Session Connection Overrides (v1.4+)
+When sessions define per-transform connection overrides (different database, file directory, or filename), the generated code uses those overrides instead of source/target defaults.
+
+### Worklet Support (v1.4+)
+Worklet workflows are detected and generate separate `run_worklet_NAME(config)` functions. The main workflow calls these automatically for Worklet task types.
+
+### Type Casting at Target Writes (v1.4+)
+Target field datatypes are mapped to pandas types and generate proper casting code:
+- Integers: nullable `Int64`/`Int32` or `fillna(0).astype(int)` for NOT NULL
+- Dates: `pd.to_datetime(errors='coerce')`
+- Decimals/Floats: `pd.to_numeric(errors='coerce')`
+- Booleans: `.astype('boolean')`
+
+### Flat File Handling (v1.3+)
+Parses FLATFILE metadata for delimiter, fixed-width, header lines, skip rows, quote/escape chars. Generates `pd.read_fwf()` for fixed-width or enriched `read_file()` for delimited.
+
+### Mapplet Inlining (v1.3+)
+Expands Mapplet instances into prefixed transforms, rewires connectors, and eliminates duplication.
+
+### Decision Tasks (v1.3+)
+Converts Informatica decision conditions to Python if/else branches with proper variable substitution.
+
+### Expression Converter (80+ Functions)
+
+Converts Informatica expressions to Python equivalents:
+
+- **String:** SUBSTR, LTRIM, RTRIM, UPPER, LOWER, LPAD, RPAD, INSTR, LENGTH, CONCAT, REPLACE, REG_EXTRACT, REG_REPLACE, REVERSE, INITCAP, CHR, ASCII
+- **Date:** ADD_TO_DATE, DATE_DIFF, GET_DATE_PART, SYSDATE, SYSTIMESTAMP, TO_DATE, TO_CHAR, TRUNC (date)
+- **Numeric:** ROUND, TRUNC, MOD, ABS, CEIL, FLOOR, POWER, SQRT, LOG, EXP, SIGN
+- **Conversion:** TO_INTEGER, TO_BIGINT, TO_FLOAT, TO_DECIMAL, TO_CHAR, TO_DATE
+- **Null handling:** IIF, DECODE, NVL, NVL2, ISNULL, IS_SPACES, IS_NUMBER
+- **Aggregate:** SUM, AVG, COUNT, MIN, MAX, FIRST, LAST, MEDIAN, STDDEV, VARIANCE
+- **Lookup:** :LKP expressions with dynamic lookup references
+- **Variable:** SETVARIABLE / mapping variable assignment
+
+## Requirements
+
+- Python >= 3.8
+- lxml >= 4.9.0
+- PyYAML >= 6.0
+
+## Changelog
+
+### v1.4.x (Phase 3)
+- Session connection overrides for sources and targets
+- Worklet function generation with safe invocation
+- Type casting at target writes based on TARGETFIELD datatypes
+- Flat-file session path overrides properly wired
+
+### v1.3.x (Phase 2)
+- FLATFILE metadata in source reads and target writes
+- Normalizer with `pd.melt()`
+- Rank with group-by and Top-N filtering
+- Decision tasks with real if/else branches
+- Mapplet instance inlining
+
+### v1.2.x (Phase 1)
+- Core parser for all 72 XML tags
+- Expression converter with 80+ functions
+- Aggregator, Joiner, Lookup code generation
+- Workflow orchestration with topological task ordering
+- Multi-library support (pandas, dask, polars, vaex, modin)
+
+## Development
+
+```bash
+# Clone and install in development mode
+cd informatica_python
+pip install -e ".[dev]"
+
+# Run tests (25 tests)
+pytest tests/test_converter.py -v
+```
+
+## License
+
+MIT License - Copyright (c) 2025 Nick
+
+See [LICENSE](LICENSE) for details.

informatica_python-1.5.0/informatica_python/__init__.py

@@ -0,0 +1,13 @@
+"""
+informatica-python: Convert Informatica PowerCenter workflow XML to Python/PySpark code.
+
+Copyright (c) 2025 Nick. All rights reserved.
+Licensed under the MIT License.
+"""
+
+from informatica_python.converter import InformaticaConverter
+
+__version__ = "1.5.0"
+__author__ = "Nick"
+__license__ = "MIT"
+__all__ = ["InformaticaConverter"]

{informatica_python-1.4.1 → informatica_python-1.5.0}/informatica_python/cli.py

@@ -41,6 +41,11 @@ def main():
         default=None,
         help="Save parsed JSON to a file",
     )
+    parser.add_argument(
+        "--param-file",
+        default=None,
+        help="Path to Informatica .param file for variable substitution",
+    )
 
     args = parser.parse_args()
 

{informatica_python-1.4.1 → informatica_python-1.5.0}/informatica_python/generators/helper_gen.py

@@ -44,13 +44,20 @@ def generate_helper_functions(folder: FolderDef, data_lib: str = "pandas") -> st
     lines.append('logger = logging.getLogger("informatica_converter")')
     lines.append("")
     lines.append("")
-    lines.append("def load_config(config_path='config.yml'):")
-    lines.append('    """Load configuration from YAML file."""')
+    lines.append("def load_config(config_path='config.yml', param_file=None):")
+    lines.append('    """Load configuration from YAML file, optionally merging Informatica .param file."""')
     lines.append("    with open(config_path, 'r') as f:")
-    lines.append("        return yaml.safe_load(f)")
+    lines.append("        config = yaml.safe_load(f) or {}")
+    lines.append("    if param_file:")
+    lines.append("        params = parse_param_file(param_file)")
+    lines.append("        config['params'] = params")
+    lines.append("        for key, val in params.items():")
+    lines.append("            os.environ[f'INFA_VAR_{key}'] = str(val)")
+    lines.append("    return config")
     lines.append("")
     lines.append("")
 
+    _add_param_file_functions(lines)
     _add_db_functions(lines, data_lib)
     _add_file_functions(lines, data_lib)
     _add_expression_helpers(lines)
@@ -59,6 +66,61 @@ def generate_helper_functions(folder: FolderDef, data_lib: str = "pandas") -> st
     return "\n".join(lines)
 
 
+def _add_param_file_functions(lines):
+    lines.append("# ============================================================")
+    lines.append("# Informatica Parameter File Support")
+    lines.append("# ============================================================")
+    lines.append("")
+    lines.append("")
+    lines.append("def parse_param_file(param_path):")
+    lines.append('    """')
+    lines.append("    Parse an Informatica .param file into a flat dict of variable names to values.")
+    lines.append("    Supports standard Informatica parameter file format:")
+    lines.append("      [Global]")
+    lines.append("      $$VAR_NAME=value")
+    lines.append("      [folder_name.WF:workflow_name.ST:session_name]")
+    lines.append("      $$CONN_NAME=value")
+    lines.append('    """')
+    lines.append("    params = {}")
+    lines.append("    if not os.path.exists(param_path):")
+    lines.append("        logger.warning(f'Parameter file not found: {param_path}')")
+    lines.append("        return params")
+    lines.append("")
+    lines.append("    current_section = 'Global'")
+    lines.append("    with open(param_path, 'r') as f:")
+    lines.append("        for line_num, line in enumerate(f, 1):")
+    lines.append("            line = line.strip()")
+    lines.append("            if not line or line.startswith('#'):")
+    lines.append("                continue")
+    lines.append("            if line.startswith('[') and line.endswith(']'):")
+    lines.append("                current_section = line[1:-1].strip()")
+    lines.append("                continue")
+    lines.append("            if '=' in line:")
+    lines.append("                key, _, value = line.partition('=')")
+    lines.append("                key = key.strip()")
+    lines.append("                value = value.strip()")
+    lines.append("                clean_key = key.lstrip('$')")
+    lines.append("                params[clean_key] = value")
+    lines.append("                if current_section != 'Global':")
+    lines.append("                    params[f'{current_section}.{clean_key}'] = value")
+    lines.append("    logger.info(f'Loaded {len(params)} parameters from {param_path}')")
+    lines.append("    return params")
+    lines.append("")
+    lines.append("")
+    lines.append("def get_param(config, var_name, default=''):")
+    lines.append('    """Get a parameter value from config params, then env vars, then default."""')
+    lines.append("    clean = var_name.lstrip('$')")
+    lines.append("    params = config.get('params', {})")
+    lines.append("    if clean in params:")
+    lines.append("        return params[clean]")
+    lines.append("    env_val = os.environ.get(f'INFA_VAR_{clean}')")
+    lines.append("    if env_val is not None:")
+    lines.append("        return env_val")
+    lines.append("    return default")
+    lines.append("")
+    lines.append("")
+
+
 def _add_db_functions(lines, data_lib):
     lines.append("# ============================================================")
     lines.append("# Database Operations")
@@ -1060,14 +1122,26 @@ def _add_expression_helpers(lines):
     lines.append("    return None")
     lines.append("")
     lines.append("")
-    lines.append("def get_variable(var_name):")
-    lines.append('    """Get workflow/mapping variable value."""')
-    lines.append("    return os.environ.get(f'INFA_VAR_{var_name}', '')")
+    lines.append("_param_store = {}")
+    lines.append("")
+    lines.append("")
+    lines.append("def get_variable(var_name, config=None):")
+    lines.append('    """Get workflow/mapping variable value from params, env vars, or param store."""')
+    lines.append("    clean = var_name.lstrip('$')")
+    lines.append("    if config and 'params' in config:")
+    lines.append("        val = config['params'].get(clean)")
+    lines.append("        if val is not None:")
+    lines.append("            return val")
+    lines.append("    if clean in _param_store:")
+    lines.append("        return _param_store[clean]")
+    lines.append("    return os.environ.get(f'INFA_VAR_{clean}', '')")
     lines.append("")
     lines.append("")
     lines.append("def set_variable(var_name, value):")
     lines.append('    """Set workflow/mapping variable value."""')
-    lines.append("    os.environ[f'INFA_VAR_{var_name}'] = str(value)")
+    lines.append("    clean = var_name.lstrip('$')")
+    lines.append("    _param_store[clean] = value")
+    lines.append("    os.environ[f'INFA_VAR_{clean}'] = str(value)")
     lines.append("    return value")
     lines.append("")
     lines.append("")