src-connectors 0.1.0__tar.gz

src_connectors-0.1.0/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.4
+Name: src-connectors
+Version: 0.1.0
+Summary: High-performance data connectors for SQL Server, Spark, Iceberg, and JDBC.
+Author: canh.ld
+Requires-Python: >=3.11,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Provides-Extra: all
+Provides-Extra: dev
+Provides-Extra: docs
+Provides-Extra: oracle
+Provides-Extra: postgres
+Provides-Extra: spark
+Provides-Extra: sqlserver
+Provides-Extra: trino
+Requires-Dist: mkdocs (>=1.6.0,<2.0.0) ; extra == "docs"
+Requires-Dist: mkdocs-material (>=9.6.0,<10.0.0) ; extra == "docs"
+Requires-Dist: mypy (>=1.17.0,<2.0.0) ; extra == "dev"
+Requires-Dist: numpy (>=1.24.0) ; extra == "spark"
+Requires-Dist: oracledb (>=3.2.0,<4.0.0) ; extra == "all"
+Requires-Dist: oracledb (>=3.2.0,<4.0.0) ; extra == "oracle"
+Requires-Dist: pandas (>=2.0.0) ; extra == "spark"
+Requires-Dist: pandas (>=2.3.0,<3.0.0)
+Requires-Dist: pre-commit (>=4.3.0,<5.0.0) ; extra == "dev"
+Requires-Dist: psycopg[binary] (>=3.2.0,<4.0.0) ; extra == "all"
+Requires-Dist: psycopg[binary] (>=3.2.0,<4.0.0) ; extra == "postgres"
+Requires-Dist: pyarrow (>=14.0.0) ; extra == "spark"
+Requires-Dist: pyarrow (>=21.0.0,<22.0.0)
+Requires-Dist: pydantic (>=2.11.0,<3.0.0)
+Requires-Dist: pyodbc (==5.3.0) ; extra == "all"
+Requires-Dist: pyodbc (==5.3.0) ; extra == "sqlserver"
+Requires-Dist: pyspark (>=3.5.0,<4.0.0) ; extra == "all"
+Requires-Dist: pyspark (>=3.5.0,<4.0.0) ; extra == "spark"
+Requires-Dist: pytest (>=9.0.3,<10.0.0) ; extra == "dev"
+Requires-Dist: pytest-cov (>=6.2.0,<7.0.0) ; extra == "dev"
+Requires-Dist: python-dotenv (>=1.2.2,<2.0.0)
+Requires-Dist: requests (>=2.33.1,<3.0.0)
+Requires-Dist: ruff (>=0.12.0,<1.0.0) ; extra == "dev"
+Requires-Dist: sqlalchemy (==2.0.49) ; extra == "sqlserver"
+Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0) ; extra == "all"
+Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0) ; extra == "oracle"
+Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0) ; extra == "postgres"
+Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0) ; extra == "trino"
+Requires-Dist: structlog (>=25.4.0,<26.0.0)
+Requires-Dist: tenacity (>=9.1.0,<10.0.0)
+Requires-Dist: tqdm (>=4.66.0,<5.0.0)
+Requires-Dist: trino (>=0.336.0,<1.0.0) ; extra == "all"
+Requires-Dist: trino (>=0.336.0,<1.0.0) ; extra == "trino"
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://img.icons8.com/dt/200/database-connectivity.png" alt="src-connectors Logo" width="160" />
+</p>
+
+<h1 align="center">src-connectors</h1>
+
+<p align="center">
+  <strong>The ultimate library for high-speed data connectivity, querying, and reading.</strong>
+</p>
+
+<p align="center">
+  <a href="#"><img src="https://img.shields.io/badge/version-0.1.0-blue.svg" alt="Version"></a>
+  <a href="#"><img src="https://img.shields.io/badge/python-3.11+-green.svg" alt="Python Versions"></a>
+  <a href="#"><img src="https://img.shields.io/badge/license-MIT-orange.svg" alt="License"></a>
+</p>
+
+---
+
+## 📖 What is src-connectors?
+
+**src-connectors** is a professional-grade Python library designed to simplify the complexity of connecting to, querying, and reading data from multiple sources. It provides a standardized, high-performance interface for data engineers and analysts to pull data into their applications without worrying about underlying driver intricacies or memory management.
+
+Whether you are fetching a small sample for exploration or streaming billions of rows for a production pipeline, **src-connectors** ensures your data access is reliable, secure, and fast.
+
+**Current Support:**
+- **SQL Server**: Robust connectivity via `pyodbc` with support for high-speed batched reads.
+- **Apache Spark**: Modular engine setup supporting **Iceberg** catalogs (Hadoop/REST/S3) and **JDBC** data sources.
+
+---
+
+## 🚀 Getting Started
+
+### Installation
+
+Install `src-connectors` utilizing `pip` or `Poetry`. Choose your extras based on the platforms you need to query:
+
+```bash
+# Using pip for a specific extra
+pip install "src-connectors[spark,sqlserver]"
+
+# Using Poetry (Recommended)
+poetry add "src-connectors[all]"
+```
+
+### Quick Start
+
+1. **Querying SQL Server**: Fetch data directly into a DataFrame.
+```python
+from src_connectors import SQLServerConnector
+
+connector = SQLServerConnector()
+# Reading data into a pandas DataFrame
+df = connector.execute_query("SELECT TOP 10 * FROM orders", output_type="dataframe")
+```
+
+2. **Reading from Iceberg (Spark Engine)**: Standardized data access for big data.
+```python
+from src_connectors import SparkConnector
+
+# Initialize and configure the Iceberg catalog
+connector = SparkConnector(spark_master="local[*]")
+connector.configure_iceberg({"iceberg_warehouse": "prod_catalog"})
+# --- Step 4: Execute Query using Spark SQL ---
+# Execute a query and fetch results
+df = connector.execute_query("SELECT * FROM prod_catalog.db.table", output_type="dataframe")
+```
+
+---
+
+## 📚 Documentation
+
+For in-depth architectural overviews, detailed configuration settings, and complex usage examples, please refer to the unified documentation in the `docs/` folder:
+
+- [Index Overview](docs/index.md)
+- [SQL Server Connector Guide](docs/sqlserver.md)
+- [Apache Spark (Iceberg & JDBC) Guide](docs/spark.md)
+
+---
+
+## 🏗 Architecture & Design
+
+The library is built on a **Modular Connector Pattern**. Every component focuses on a specific data source while sharing a common execution interface. This decoupling allows engineers to inject custom configurations without breaking the core read/query logic.
+
+<p align="center">
+  <strong>Execution Logic:</strong><br>
+  <code>Connector Initialization</code> → <code>Component Configuration</code> → <code>Unified Connection</code> → <code>Optimized Query Execution</code>
+</p>
+
+---
+
+## ✨ Key Features
+
+- **Querying Consistency**: One standard `execute_query()` method across all connectors, supporting SQL and Spark SQL.
+- **Optimized Data Reading**: Native support for returning DataFrames (Pandas/Spark), Lists of Dictionaries, or NumPy Arrays.
+- **Memory-Safe Batching**: Integrated `stream=True` functionality for reading large datasets through Python Generators to prevent memory overflow.
+- **Enterprise Configuration**: Layered settings management allowing for Environment defaults with per-query overwrites.
+- **Security-First**: Automatic protection and masking of credentials in all logs and metadata exports.
+- **Cloud-Ready Big Data**: Specialized support for Iceberg REST Catalogs, OAuth2, and MinIO/S3 compatible storage.
+
+---
+
+## 🛠 Supported Data Sources
+
+| Connector | Source | Driver/Engine | Role |
+| :--- | :--- | :--- | :--- |
+| `SQLServerConnector` | SQL Server | `pyodbc` | Query & Read |
+| `SparkConnector` | Spark / Iceberg / JDBC | `pyspark` | Query & Big Data Read |
+| `OracleConnector` | Oracle DB | `oracledb` | *Coming Soon* |
+| `PostgresConnector` | PostgreSQL | `psycopg3` | *Planned* |
+| `TrinoConnector` | Trino | `trino` | *Planned* |
+
+---
+
+## 🔮 Roadmap
+
+We are expanding **src-connectors** to become the default data access layer for all modern infrastructures:
+
+- **Federated Query Engines**: Adding Trino and Presto support for cross-catalog querying.
+- **Streaming Sinks**: Support for writing queried data into Kafka or RabbitMQ.
+- **Advanced Authentication**: Native integration with AWS Secrets Manager, Azure Key Vault, and HashiCorp Vault.
+- **Observability**: Built-in OpenTelemetry hooks to track query performance and latency.
+
+---
+
+## 🤝 Contributing & License
+
+Contributions, issues, and feature requests are welcome! Feel free to check the [issues page](#).
+
+This project is licensed under the terms of the **MIT** license.
+

src_connectors-0.1.0/README.md

@@ -0,0 +1,129 @@
+<p align="center">
+  <img src="https://img.icons8.com/dt/200/database-connectivity.png" alt="src-connectors Logo" width="160" />
+</p>
+
+<h1 align="center">src-connectors</h1>
+
+<p align="center">
+  <strong>The ultimate library for high-speed data connectivity, querying, and reading.</strong>
+</p>
+
+<p align="center">
+  <a href="#"><img src="https://img.shields.io/badge/version-0.1.0-blue.svg" alt="Version"></a>
+  <a href="#"><img src="https://img.shields.io/badge/python-3.11+-green.svg" alt="Python Versions"></a>
+  <a href="#"><img src="https://img.shields.io/badge/license-MIT-orange.svg" alt="License"></a>
+</p>
+
+---
+
+## 📖 What is src-connectors?
+
+**src-connectors** is a professional-grade Python library designed to simplify the complexity of connecting to, querying, and reading data from multiple sources. It provides a standardized, high-performance interface for data engineers and analysts to pull data into their applications without worrying about underlying driver intricacies or memory management.
+
+Whether you are fetching a small sample for exploration or streaming billions of rows for a production pipeline, **src-connectors** ensures your data access is reliable, secure, and fast.
+
+**Current Support:**
+- **SQL Server**: Robust connectivity via `pyodbc` with support for high-speed batched reads.
+- **Apache Spark**: Modular engine setup supporting **Iceberg** catalogs (Hadoop/REST/S3) and **JDBC** data sources.
+
+---
+
+## 🚀 Getting Started
+
+### Installation
+
+Install `src-connectors` utilizing `pip` or `Poetry`. Choose your extras based on the platforms you need to query:
+
+```bash
+# Using pip for a specific extra
+pip install "src-connectors[spark,sqlserver]"
+
+# Using Poetry (Recommended)
+poetry add "src-connectors[all]"
+```
+
+### Quick Start
+
+1. **Querying SQL Server**: Fetch data directly into a DataFrame.
+```python
+from src_connectors import SQLServerConnector
+
+connector = SQLServerConnector()
+# Reading data into a pandas DataFrame
+df = connector.execute_query("SELECT TOP 10 * FROM orders", output_type="dataframe")
+```
+
+2. **Reading from Iceberg (Spark Engine)**: Standardized data access for big data.
+```python
+from src_connectors import SparkConnector
+
+# Initialize and configure the Iceberg catalog
+connector = SparkConnector(spark_master="local[*]")
+connector.configure_iceberg({"iceberg_warehouse": "prod_catalog"})
+# --- Step 4: Execute Query using Spark SQL ---
+# Execute a query and fetch results
+df = connector.execute_query("SELECT * FROM prod_catalog.db.table", output_type="dataframe")
+```
+
+---
+
+## 📚 Documentation
+
+For in-depth architectural overviews, detailed configuration settings, and complex usage examples, please refer to the unified documentation in the `docs/` folder:
+
+- [Index Overview](docs/index.md)
+- [SQL Server Connector Guide](docs/sqlserver.md)
+- [Apache Spark (Iceberg & JDBC) Guide](docs/spark.md)
+
+---
+
+## 🏗 Architecture & Design
+
+The library is built on a **Modular Connector Pattern**. Every component focuses on a specific data source while sharing a common execution interface. This decoupling allows engineers to inject custom configurations without breaking the core read/query logic.
+
+<p align="center">
+  <strong>Execution Logic:</strong><br>
+  <code>Connector Initialization</code> → <code>Component Configuration</code> → <code>Unified Connection</code> → <code>Optimized Query Execution</code>
+</p>
+
+---
+
+## ✨ Key Features
+
+- **Querying Consistency**: One standard `execute_query()` method across all connectors, supporting SQL and Spark SQL.
+- **Optimized Data Reading**: Native support for returning DataFrames (Pandas/Spark), Lists of Dictionaries, or NumPy Arrays.
+- **Memory-Safe Batching**: Integrated `stream=True` functionality for reading large datasets through Python Generators to prevent memory overflow.
+- **Enterprise Configuration**: Layered settings management allowing for Environment defaults with per-query overwrites.
+- **Security-First**: Automatic protection and masking of credentials in all logs and metadata exports.
+- **Cloud-Ready Big Data**: Specialized support for Iceberg REST Catalogs, OAuth2, and MinIO/S3 compatible storage.
+
+---
+
+## 🛠 Supported Data Sources
+
+| Connector | Source | Driver/Engine | Role |
+| :--- | :--- | :--- | :--- |
+| `SQLServerConnector` | SQL Server | `pyodbc` | Query & Read |
+| `SparkConnector` | Spark / Iceberg / JDBC | `pyspark` | Query & Big Data Read |
+| `OracleConnector` | Oracle DB | `oracledb` | *Coming Soon* |
+| `PostgresConnector` | PostgreSQL | `psycopg3` | *Planned* |
+| `TrinoConnector` | Trino | `trino` | *Planned* |
+
+---
+
+## 🔮 Roadmap
+
+We are expanding **src-connectors** to become the default data access layer for all modern infrastructures:
+
+- **Federated Query Engines**: Adding Trino and Presto support for cross-catalog querying.
+- **Streaming Sinks**: Support for writing queried data into Kafka or RabbitMQ.
+- **Advanced Authentication**: Native integration with AWS Secrets Manager, Azure Key Vault, and HashiCorp Vault.
+- **Observability**: Built-in OpenTelemetry hooks to track query performance and latency.
+
+---
+
+## 🤝 Contributing & License
+
+Contributions, issues, and feature requests are welcome! Feel free to check the [issues page](#).
+
+This project is licensed under the terms of the **MIT** license.

src_connectors-0.1.0/pyproject.toml

@@ -0,0 +1,107 @@
+[project]
+name = "src-connectors"
+version = "0.1.0"
+description = "High-performance data connectors for SQL Server, Spark, Iceberg, and JDBC."
+authors = [
+    { name = "canh.ld" }
+]
+readme = "README.md"
+requires-python = ">=3.11,<4.0"
+
+dependencies = [
+    "requests>=2.33.1,<3.0.0",
+    "python-dotenv>=1.2.2,<2.0.0",
+    "tqdm>=4.66.0,<5.0.0",
+
+    # Data handling
+    "pandas>=2.3.0,<3.0.0",
+    "pyarrow>=21.0.0,<22.0.0",
+
+    # Validation / config
+    "pydantic>=2.11.0,<3.0.0",
+
+    # Retry / resiliency
+    "tenacity>=9.1.0,<10.0.0",
+
+    # Logging
+    "structlog>=25.4.0,<26.0.0"
+]
+
+[tool.poetry]
+# Giữ cấu trúc flat layout để nhận diện các module ngang hàng root
+packages = [
+    { include = "src_connectors" },
+    { include = "variables" },
+    { include = "common" }
+]
+
+[project.optional-dependencies]
+postgres = [
+    "psycopg[binary]>=3.2.0,<4.0.0",
+    "sqlalchemy>=2.0.0,<3.0.0"
+]
+
+trino = [
+    "trino>=0.336.0,<1.0.0",
+    "sqlalchemy>=2.0.0,<3.0.0"
+]
+
+sqlserver = [
+    "pyodbc==5.3.0",
+    "sqlalchemy==2.0.49"
+]
+
+oracle = [
+    "oracledb>=3.2.0,<4.0.0",
+    "sqlalchemy>=2.0.0,<3.0.0"
+]
+
+spark = [
+    "pyspark>=3.5.0,<4.0.0",
+    "pandas>=2.0.0",
+    "pyarrow>=14.0.0",
+    "numpy>=1.24.0"
+]
+
+# Định nghĩa group 'all' để lệnh cài đặt ban đầu không bị báo WARNING
+all = [
+    "psycopg[binary]>=3.2.0,<4.0.0",
+    "trino>=0.336.0,<1.0.0",
+    "pyodbc==5.3.0",
+    "oracledb>=3.2.0,<4.0.0",
+    "pyspark>=3.5.0,<4.0.0",
+    "sqlalchemy>=2.0.0,<3.0.0"
+]
+
+dev = [
+    "pytest>=9.0.3,<10.0.0",
+    "pytest-cov>=6.2.0,<7.0.0",
+    "ruff>=0.12.0,<1.0.0",
+    "mypy>=1.17.0,<2.0.0",
+    "pre-commit>=4.3.0,<5.0.0"
+]
+
+docs = [
+    "mkdocs>=1.6.0,<2.0.0",
+    "mkdocs-material>=9.6.0,<10.0.0"
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.pytest.ini_options]
+pythonpath = [
+    "."
+]
+testpaths = [
+    "tests"
+]
+
+[tool.mypy]
+python_version = "3.11"
+ignore_missing_imports = true
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"

src_connectors-0.1.0/src_connectors/__init__.py

@@ -0,0 +1,18 @@
+import sys
+from typing import Any
+
+__all__ = ["SQLServerConnector", "SparkConnector"]
+
+
+def __getattr__(name: str) -> Any:
+    if name == "SQLServerConnector":
+        from src_connectors.src_sqlserver.connector import SQLServerConnector
+
+        return SQLServerConnector
+
+    if name == "SparkConnector":
+        from src_connectors.src_spark.connector import SparkConnector
+
+        return SparkConnector
+
+    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")

src_connectors-0.1.0/src_connectors/src_base/__init__.py

@@ -0,0 +1,3 @@
+from .connector import BaseConnector
+
+__all__ = ["BaseConnector"]

src_connectors-0.1.0/src_connectors/src_base/connector.py

@@ -0,0 +1,31 @@
+from abc import ABC, abstractmethod
+from typing import Any, Optional
+
+class BaseConnector(ABC):
+    """
+    Abstract base class for database connectors.
+    """
+
+    @abstractmethod
+    def connect(self) -> Any:
+        """Establish a connection to the database."""
+        pass
+
+    @abstractmethod
+    def close(self) -> None:
+        """Close the database connection."""
+        pass
+
+    @abstractmethod
+    def execute_query(
+        self,
+        query: str,
+        params: Optional[tuple] = None,
+        stream: bool = False,
+        batch_size: int = 1000,
+        output_type: str = "list",
+        **kwargs: Any
+    ) -> Any:
+        """Execute a query and return the results."""
+        pass
+

src_connectors-0.1.0/src_connectors/src_spark/__init__.py

@@ -0,0 +1,6 @@
+from src_connectors.src_spark.connector import SparkConnector
+from src_connectors.src_spark.configs.core_config import SparkCoreConfig
+from src_connectors.src_spark.configs.iceberg import SparkIcebergConfig
+from src_connectors.src_spark.configs.jdbc import SparkJdbcConfig
+
+__all__ = ["SparkConnector", "SparkCoreConfig", "SparkIcebergConfig", "SparkJdbcConfig"]

src_connectors-0.1.0/src_connectors/src_spark/configs/__init__.py

@@ -0,0 +1,6 @@
+from src_connectors.src_spark.configs.core_config import SparkCoreConfig
+from src_connectors.src_spark.configs.iceberg import SparkIcebergConfig
+from src_connectors.src_spark.configs.jdbc import SparkJdbcConfig
+from src_connectors.src_spark.configs.base import SparkBaseComponent
+
+__all__ = ["SparkCoreConfig", "SparkIcebergConfig", "SparkJdbcConfig", "SparkBaseComponent"]

src_connectors-0.1.0/src_connectors/src_spark/configs/base.py

@@ -0,0 +1,30 @@
+from abc import ABC, abstractmethod
+from typing import List, Dict
+
+class SparkBaseComponent(ABC):
+    """
+    Abstract base class for Spark components.
+
+    This class defines the interface for components that require
+    specific Spark configurations and external packages.
+    """
+
+    @abstractmethod
+    def get_spark_config(self) -> Dict:
+        """
+        Return Spark configuration settings.
+
+        Returns:
+            dict: A dictionary of Spark configuration keys and values.
+        """
+        pass
+
+    @abstractmethod
+    def get_required_spark_packages(self) -> List:
+        """
+        Return a list of required Spark packages.
+
+        Returns:
+            list: A list of Maven coordinates or package names.
+        """
+        pass

src_connectors-0.1.0/src_connectors/src_spark/configs/core_config.py

@@ -0,0 +1,138 @@
+from typing import Dict, List, Any, Optional
+from pydantic import BaseModel, Field
+
+from src_connectors.src_spark.configs.base import SparkBaseComponent
+from variables.spark import SparkVariables
+
+# Load environment-based defaults
+_spark_defaults = SparkVariables.get_spark_base_config()
+
+
+class SparkCoreConfig(BaseModel, SparkBaseComponent):
+    """
+    Configuration model for core Apache Spark engine settings.
+
+    This class manages Spark core parameters including resource allocation (memory, cores),
+    cluster connectivity, and runtime dependencies. Defaults are initialized from
+    environment variables.
+    """
+
+    # Cluster & Application Settings
+    spark_master: str = Field(
+        default=_spark_defaults.get("SPARK_MASTER") or "local[*]",
+        description="The Spark master URL to connect to."
+    )
+    spark_driver_host: str = Field(
+        default=_spark_defaults.get("SPARK_DRIVER_HOST") or "127.0.0.1",
+        description="Hostname or IP address for the driver to listen on."
+    )
+    spark_driver_bind_address: str = Field(
+        default=_spark_defaults.get("SPARK_DRIVER_BIND_ADDRESS") or "127.0.0.1",
+        description="Address to bind to for the driver."
+    )
+    spark_app_name: str = Field(
+        default=_spark_defaults.get("SPARK_APP_NAME") or "SparkApp",
+        description="The name of the Spark application."
+    )
+
+    # Resource Management
+    spark_executor_memory: str = Field(
+        default=_spark_defaults.get("SPARK_EXECUTOR_MEMORY") or "2g",
+        description="Amount of memory to use per executor process."
+    )
+    spark_executor_cores: int = Field(
+        default=int(_spark_defaults.get("SPARK_EXECUTOR_CORES") or 2),
+        description="The number of cores to use on each executor."
+    )
+    spark_driver_memory: str = Field(
+        default=_spark_defaults.get("SPARK_DRIVER_MEMORY") or "4g",
+        description="Amount of memory to use for the driver process."
+    )
+    spark_driver_cores: int = Field(
+        default=int(_spark_defaults.get("SPARK_DRIVER_CORES") or 2),
+        description="Number of cores to use for the driver process."
+    )
+
+    # Dependencies & Runtime
+    spark_jars: Optional[str] = Field(
+        default=_spark_defaults.get("SPARK_LOCAL_JARS") or "",
+        description="Comma-separated local jar paths."
+    )
+    spark_jars_packages: Optional[str] = Field(
+        default=_spark_defaults.get("SPARK_JARS_PACKAGES") or "",
+        description="Comma-separated Maven coordinates of jars to include."
+    )
+    spark_jars_ivy: str = Field(
+        default=_spark_defaults.get("SPARK_JARS_IVY") or "/tmp/.ivy2",
+        description="Path to local Ivy repository for dependency resolution."
+    )
+
+    # Environment & Versioning
+    spark_version: str = Field(
+        default=_spark_defaults.get("SPARK_VERSION") or "3.0.0",
+        description="Target Spark version."
+    )
+    spark_minor_version: str = Field(
+        default=_spark_defaults.get("SPARK_MINOR_VERSION") or "3.0",
+        description="Target Spark minor version."
+    )
+    spark_local_dir: str = Field(
+        default=_spark_defaults.get("SPARK_LOCAL_DIR") or "/tmp/spark",
+        description="Directory for Spark 'scratch' space (map outputs, RDD spills)."
+    )
+
+    def get_spark_config(self) -> Dict[str, Any]:
+        """
+        Generates a dictionary of Spark configuration properties.
+
+        Returns:
+            Dict[str, Any]: Flat dictionary of 'spark.*' configuration keys and values.
+        """
+        config = {
+            "spark.master": self.spark_master,
+            "spark.app.name": self.spark_app_name,
+            "spark.executor.memory": self.spark_executor_memory,
+            "spark.executor.cores": self.spark_executor_cores,
+            "spark.driver.memory": self.spark_driver_memory,
+            "spark.driver.cores": self.spark_driver_cores,
+            "spark.jars.ivy": self.spark_jars_ivy,
+            "spark.local.dir": self.spark_local_dir,
+        }
+
+        # Handle Driver Host logic:
+        # Force 127.0.0.1 ONLY if running in local mode to avoid RPC errors on dev machines.
+        # In cluster mode (K8s/YARN), we let Spark/Cluster Manager handle it or use provided host.
+        is_local = self.spark_master.startswith("local")
+        if is_local:
+            config["spark.driver.host"] = self.spark_driver_host
+            config["spark.driver.bindAddress"] = self.spark_driver_bind_address
+        else:
+            # In cluster mode, only apply if explicitly provided (not default 127.0.0.1)
+            if self.spark_driver_host != "127.0.0.1":
+                config["spark.driver.host"] = self.spark_driver_host
+            if self.spark_driver_bind_address != "127.0.0.1":
+                config["spark.driver.bindAddress"] = self.spark_driver_bind_address
+
+        # Handle Jars and Packages logic priority
+        if self.spark_jars:
+            config["spark.jars"] = self.spark_jars
+        elif self.spark_jars_packages:
+            config["spark.jars.packages"] = self.spark_jars_packages
+
+        return config
+
+    def get_required_spark_packages(self) -> List[str]:
+        """
+        Returns a list of required Spark packages.
+        If local jars are provided, packages are usually skipped to avoid conflicts.
+
+        Returns:
+            List[str]: A list of cleaned Maven package coordinates.
+        """
+        if self.spark_jars:
+            return []
+
+        if not self.spark_jars_packages:
+            return []
+
+        return [pkg.strip() for pkg in self.spark_jars_packages.split(",") if pkg.strip()]