overture-spark 0.1.0__tar.gz

overture_spark-0.1.0/.python-version

@@ -0,0 +1 @@
+3.11

overture_spark-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Overture Maps
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

overture_spark-0.1.0/PKG-INFO

@@ -0,0 +1,74 @@
+Metadata-Version: 2.4
+Name: overture-spark
+Version: 0.1.0
+Summary: Portable, framework-agnostic Spark job runtime helpers.
+Author: Overture Maps Foundation
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/OvertureMaps/overture-core
+Project-URL: Source, https://github.com/OvertureMaps/overture-core/tree/main/packages/overture_spark
+Project-URL: Issues, https://github.com/OvertureMaps/overture-core/issues
+Keywords: overture,overturemaps,spark
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: sql-spark
+Requires-Dist: pyspark==3.5.0; extra == "sql-spark"
+Requires-Dist: apache-sedona==1.7.0; extra == "sql-spark"
+Requires-Dist: moto[server]>=5.0.25; extra == "sql-spark"
+Provides-Extra: dev
+Requires-Dist: pytest>=9.1.1; extra == "dev"
+Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
+Dynamic: license-file
+
+# overture-spark
+
+[![PyPI](https://img.shields.io/pypi/v/overture-spark.svg)](https://pypi.org/project/overture-spark/)
+[![Python versions](https://img.shields.io/pypi/pyversions/overture-spark.svg)](https://pypi.org/project/overture-spark/)
+
+Portable, framework-agnostic runtime helpers for Overture's Spark jobs. Moved from `tf-data-platform`'s `overture_spark` module (see [ops-team#535](https://github.com/OvertureMaps/ops-team/issues/535), [ops-team#536](https://github.com/OvertureMaps/ops-team/issues/536)); other `overture_spark` modules follow in later PRs (see [ops-team#532](https://github.com/OvertureMaps/ops-team/issues/532)).
+
+It contains:
+- `SparkSedonaJob` (in `job.py`) — cluster-side base class for jobs: platform detection, parameter parsing, logging, and the Sedona `SparkSession` lifecycle. Subclass it and implement `execute_job()`.
+- `test_area` — helpers for a job's optional `test_area` param (a bbox or WKT polygon that scopes a dev/test run to a small region instead of the full planet).
+
+## Writing a job
+
+```python
+from overture_spark.job import SparkSedonaJob
+
+
+class CollectionJob(SparkSedonaJob):
+    def execute_job(self) -> None:
+        path = self.get_param("input_path")
+        self.log(f"Processing {path}")
+        df = self.apply_test_area_filter(self.spark.read.parquet(path))
+        # ... your logic
+```
+
+`getSparkSedonaSession` (in `overture_spark/__init__.py`) is the one call in this package that actually needs a real Spark/Sedona session — it, and everything downstream of it, imports `pyspark`/`apache-sedona` lazily and only there. Platform detection, version/JAR helpers, and `SparkSedonaJob`'s parameter/logging logic are plain Python and importable without either installed.
+
+## Testing locally
+
+```bash
+cd packages/overture_spark
+uv sync --extra dev
+uv run pytest -v -m "not spark"
+```
+
+Tests marked `@pytest.mark.spark` need a real Spark/Sedona session (a JVM, plus Sedona's native JAR resolution), so they're excluded from the routine `pytest` run above and from `overture-spark[dev]`. Install the `sql-spark` extra to run them:
+
+```bash
+uv sync --extra dev --extra sql-spark
+uv run pytest -v --cov --cov-fail-under=95
+```
+
+CI mirrors this split: the routine `Test Python` workflow runs `pytest -m "not spark"` for this package (no coverage gate, since the Spark-only code paths aren't exercised), and the `SQL Engines` workflow's `spark (overture_spark)` job installs the `sql-spark` extra and Java, then runs the full suite with the 95% coverage floor.
+
+## Publishing
+
+See [`PACKAGE_VERSIONING.md`](../PACKAGE_VERSIONING.md) for how a version bump here turns into a PyPI release.

overture_spark-0.1.0/README.md

@@ -0,0 +1,47 @@
+# overture-spark
+
+[![PyPI](https://img.shields.io/pypi/v/overture-spark.svg)](https://pypi.org/project/overture-spark/)
+[![Python versions](https://img.shields.io/pypi/pyversions/overture-spark.svg)](https://pypi.org/project/overture-spark/)
+
+Portable, framework-agnostic runtime helpers for Overture's Spark jobs. Moved from `tf-data-platform`'s `overture_spark` module (see [ops-team#535](https://github.com/OvertureMaps/ops-team/issues/535), [ops-team#536](https://github.com/OvertureMaps/ops-team/issues/536)); other `overture_spark` modules follow in later PRs (see [ops-team#532](https://github.com/OvertureMaps/ops-team/issues/532)).
+
+It contains:
+- `SparkSedonaJob` (in `job.py`) — cluster-side base class for jobs: platform detection, parameter parsing, logging, and the Sedona `SparkSession` lifecycle. Subclass it and implement `execute_job()`.
+- `test_area` — helpers for a job's optional `test_area` param (a bbox or WKT polygon that scopes a dev/test run to a small region instead of the full planet).
+
+## Writing a job
+
+```python
+from overture_spark.job import SparkSedonaJob
+
+
+class CollectionJob(SparkSedonaJob):
+    def execute_job(self) -> None:
+        path = self.get_param("input_path")
+        self.log(f"Processing {path}")
+        df = self.apply_test_area_filter(self.spark.read.parquet(path))
+        # ... your logic
+```
+
+`getSparkSedonaSession` (in `overture_spark/__init__.py`) is the one call in this package that actually needs a real Spark/Sedona session — it, and everything downstream of it, imports `pyspark`/`apache-sedona` lazily and only there. Platform detection, version/JAR helpers, and `SparkSedonaJob`'s parameter/logging logic are plain Python and importable without either installed.
+
+## Testing locally
+
+```bash
+cd packages/overture_spark
+uv sync --extra dev
+uv run pytest -v -m "not spark"
+```
+
+Tests marked `@pytest.mark.spark` need a real Spark/Sedona session (a JVM, plus Sedona's native JAR resolution), so they're excluded from the routine `pytest` run above and from `overture-spark[dev]`. Install the `sql-spark` extra to run them:
+
+```bash
+uv sync --extra dev --extra sql-spark
+uv run pytest -v --cov --cov-fail-under=95
+```
+
+CI mirrors this split: the routine `Test Python` workflow runs `pytest -m "not spark"` for this package (no coverage gate, since the Spark-only code paths aren't exercised), and the `SQL Engines` workflow's `spark (overture_spark)` job installs the `sql-spark` extra and Java, then runs the full suite with the 95% coverage floor.
+
+## Publishing
+
+See [`PACKAGE_VERSIONING.md`](../PACKAGE_VERSIONING.md) for how a version bump here turns into a PyPI release.

overture_spark-0.1.0/overture_spark/__init__.py

@@ -0,0 +1,300 @@
+"""Cluster-side Spark/Sedona runtime for Overture jobs.
+
+Installed and imported **on** the Spark cluster (Glue / Databricks / Wherobots /
+local) and runs **inside** the executor. This is deliberately separate from the
+Airflow orchestration layer:
+
+- Orchestration (launch, cluster sizing, asset staging) lives in the external,
+  Overture-agnostic ``overture-airflow-provider``.
+- This package is the cluster-execution-time side: it builds the configured
+  Sedona ``SparkSession`` (``getSparkSedonaSession``), resolves Sedona/GeoTools
+  JAR coordinates (``SparkSedona``), and detects the platform
+  (``SparkPlatform.autodetect``). ``SparkSedonaJob`` (in ``job.py``) is the base
+  class user jobs subclass; the provider's runners import those job classes and
+  call ``run()``.
+
+Rule of thumb: launches a job via Airflow/AWS APIs -> provider/factory; runs inside
+the Spark job -> here.
+
+``pyspark``/``apache-sedona`` are only imported lazily, inside
+``getSparkSedonaSession()``, the one function that actually builds a Spark
+session. Everything else in this module (platform detection, version/JAR
+helpers) is plain Python, so it — and the tests that only exercise it — never
+pays the JVM cost of installing/starting Spark. Callers that do need a real
+session install the ``sql-spark`` extra.
+"""
+
+import os
+import sys
+from enum import IntEnum, auto
+from importlib.util import find_spec
+from typing import TYPE_CHECKING, Dict, List
+
+if TYPE_CHECKING:
+    from pyspark.sql import SparkSession
+
+
+class PackageNotFoundError(Exception):
+    def __init__(self, package_name):
+        super().__init__(
+            f"Package '{package_name}' is not installed or could not be found."
+        )
+        self.package_name = package_name
+
+
+def get_package_version(package_name):
+    from importlib.metadata import PackageNotFoundError as _DistNotFound
+    from importlib.metadata import version
+
+    try:
+        return version(package_name)
+    except _DistNotFound:
+        raise PackageNotFoundError(package_name)
+
+
+class SparkPlatform(IntEnum):
+    GLUE = auto()
+    DATABRICKS = auto()
+    WHEROBOTS = auto()
+    LOCAL = auto()
+
+    @classmethod
+    def autodetect(cls):
+        if cls.isRunningInGlue():
+            return SparkPlatform.GLUE
+        elif cls.isRunningInDatabricks():
+            return SparkPlatform.DATABRICKS
+        elif cls.isRunningInWherobots():
+            return SparkPlatform.WHEROBOTS
+        else:
+            return SparkPlatform.LOCAL
+
+    @classmethod
+    def from_str(cls, name):
+        # Normalize the input name to uppercase
+        normalized_name = name.upper()
+        if normalized_name in cls.__members__:
+            return cls[normalized_name]
+        raise ValueError(f"{name} is not a valid platform")
+
+    @classmethod
+    def isRunningInDatabricks(cls):
+        if "DATABRICKS_RUNTIME_VERSION" not in os.environ:
+            return False
+        return find_spec("pyspark.dbutils") is not None
+
+    @classmethod
+    def isRunningInDatabricksNotebook(cls):
+        # dbutils is injected into the *notebook's* execution namespace
+        # (__main__ when run through Databricks' REPL), not into whichever
+        # module happens to define this classmethod. A bare `dbutils` name
+        # lookup here only ever resolves against overture_spark's own
+        # globals, so it always raises NameError and always returns False,
+        # even inside a real Databricks notebook.
+        main = sys.modules.get("__main__")
+        return main is not None and hasattr(main, "dbutils")
+
+    @classmethod
+    def isRunningInGlue(cls):
+        try:
+            return find_spec("awsglue.utils") is not None
+        except ModuleNotFoundError:
+            return False
+
+    @classmethod
+    def isRunningInWherobots(cls):
+        marker_env_vars = (
+            "WHEROBOTS_RUNTIME_VERSION",
+            "WHEROBOTS_RUNTIME",
+            "WHEROBOTS_ENV",
+            "WHEROBOTS_CLUSTER_ID",
+            "WHEROBOTS_JOB_ID",
+        )
+        if any(os.getenv(var) for var in marker_env_vars):
+            return True
+
+        marker_paths = (
+            "/home/wherobots/run-scripts/run_submit.py",
+            "/opt/conda/envs/wherobots",
+        )
+        if any(os.path.exists(path) for path in marker_paths):
+            return True
+
+        if "wherobots" in (sys.prefix or "").lower():
+            return True
+
+        try:
+            return find_spec("wherobots.db") is not None
+        except ModuleNotFoundError:
+            return False
+
+
+class SparkSedona:
+    @classmethod
+    def getPySparkVersion(cls) -> str:
+        return get_package_version("pyspark")
+
+    @classmethod
+    def getSedonaVersion(cls) -> str:
+        return get_package_version("apache-sedona")
+
+    @classmethod
+    def getSparkVersionForSedona(
+        cls, py_spark_version: str, sedona_version: str
+    ) -> str:
+        # https://sedona.apache.org/latest-snapshot/setup/install-python/#prepare-sedona-spark-jar
+        spark_v = py_spark_version or cls.getPySparkVersion()
+        sparkMajorVersion, sparkMinorVersion = spark_v.split(".")[:2]
+        sedona_v = sedona_version or cls.getSedonaVersion()
+        sedonaMajorVersion, sedonaMinorVersion = sedona_v.split(".")[:2]
+        if sparkMajorVersion != "3":
+            raise RuntimeError("I'm only supporting spark 3")
+        if (
+            int(sparkMinorVersion) <= 3
+            and int(sedonaMajorVersion) == 1
+            and int(sedonaMinorVersion) <= 6
+        ):
+            sparkMajorMinorVersion = f"{sparkMajorVersion}.0"
+        else:
+            sparkMajorMinorVersion = f"{sparkMajorVersion}.{sparkMinorVersion}"
+        return sparkMajorMinorVersion
+
+    @classmethod
+    def getGeotoolsWrapperVersion(cls, sedona_version: str) -> str:
+        # see https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/
+        geotoolsVersionMap = {
+            "1.5.3": "28.2",
+            "1.6.1": "28.2",
+            "1.7.0": "28.5",
+            "1.7.1": "28.5",
+            "1.7.2": "28.5",
+            "1.8.0": "33.1",
+            "1.8.1": "33.1",
+        }
+        return geotoolsVersionMap[sedona_version]
+
+    @classmethod
+    def getSedonaJarPackages(
+        cls,
+        spark_platform: SparkPlatform,
+        scala_version: str,
+        py_spark_version: str,
+        sedona_version: str,
+    ) -> List[str]:
+        if spark_platform == SparkPlatform.DATABRICKS:
+            # on databricks, sedona packages need to be configured in cluster setup
+            return []
+
+        suffix = "-shaded" if spark_platform != SparkPlatform.LOCAL else ""
+        packages = [
+            f"org.apache.sedona:sedona-spark{suffix}-{cls.getSparkVersionForSedona(py_spark_version=py_spark_version, sedona_version=sedona_version)}_{scala_version}:{sedona_version}",
+            f"org.datasyslab:geotools-wrapper:{sedona_version}-{cls.getGeotoolsWrapperVersion(sedona_version=sedona_version)}",
+        ]
+        return packages
+
+
+def getSparkSedonaSession(
+    spark_platform: SparkPlatform = None,
+    app_name: str = "SparkSedonaApp",
+    extra_spark_conf: Dict[str, str] = None,
+    extra_packages: List[str] = None,
+) -> "SparkSession":
+    # Lazy: this is the one call site that actually needs pyspark/sedona
+    # installed and starts a JVM, see the module docstring.
+    from pyspark.sql import SparkSession
+    from sedona.spark import KryoSerializer, SedonaContext, SedonaKryoRegistrator
+
+    spark_platform = spark_platform or SparkPlatform.autodetect()
+    extra_spark_conf = extra_spark_conf or {}
+    extra_packages = extra_packages or []
+    if (
+        spark_platform == SparkPlatform.DATABRICKS
+        and SparkPlatform.isRunningInDatabricksNotebook()
+    ):
+        global spark
+        if len(extra_spark_conf) > 0:
+            raise RuntimeError(
+                "You need to set extra spark configuration when configuring the databricks cluster"
+            )
+    else:
+        # init a new spark session
+        builder = SparkSession.builder.appName(app_name)
+        if spark_platform == SparkPlatform.LOCAL:
+            hadoopAwsHelper = HadoopAwsInstaller()
+            jars = (
+                SparkSedona.getSedonaJarPackages(
+                    spark_platform=spark_platform,
+                    py_spark_version=SparkSedona.getPySparkVersion(),
+                    sedona_version=SparkSedona.getSedonaVersion(),
+                    scala_version="2.12",
+                )
+                + hadoopAwsHelper.install()
+                + extra_packages
+            )
+            (
+                builder.master("local[*]")  # * = use all available cores
+                .config("spark.serializer", KryoSerializer.getName)
+                .config("spark.kryo.registrator", SedonaKryoRegistrator.getName)
+                .config(
+                    "spark.jars.repositories",
+                    "https://artifacts.unidata.ucar.edu/repository/unidata-all",
+                )
+                .config("spark.jars.packages", ",".join(jars))
+                .config(
+                    "spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"
+                )
+            )
+        else:  # pragma: no cover - non-local platforms need a real cluster to exercise
+            # sedona jars for other (non-local) spark platforms are configured in SparkSedonaOperator*
+            pass
+        builder.config("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
+        for k, v in extra_spark_conf.items():
+            builder.config(k, v)
+        spark = builder.getOrCreate()
+
+    spark = SedonaContext.create(spark)
+    is_glue = spark_platform == SparkPlatform.GLUE
+    if is_glue:  # pragma: no cover - needs a real Glue job runtime
+        from awsglue.context import GlueContext
+
+        spark = GlueContext(spark).spark_session
+    return spark
+
+
+def isLocalSpark(spark):
+    return spark.conf.get("spark.master").startswith("local")
+
+
+class HadoopAwsInstaller:
+    def __init__(self):
+        # Resolve pyspark's own install location via its module spec instead
+        # of site.getsitepackages(), which only looks at the interpreter's
+        # site-packages directories and returns nothing outside a venv (e.g.
+        # a system-Python install, or pyspark installed via --user). That
+        # left _getInstalledVersion() unable to find the bundled jars, and
+        # install() silently built an invalid "hadoop-aws:None" coordinate.
+        pyspark_spec = find_spec("pyspark")
+        if pyspark_spec and pyspark_spec.submodule_search_locations:
+            self.jarPath = os.path.join(
+                pyspark_spec.submodule_search_locations[0], "jars"
+            )
+        else:
+            self.jarPath = None
+
+    def install(self):
+        hadoopVersion = self._getInstalledVersion()
+        if not hadoopVersion:
+            raise RuntimeError(
+                "Could not determine the installed Hadoop version from "
+                "pyspark's bundled jars; is pyspark installed?"
+            )
+        return [f"org.apache.hadoop:hadoop-aws:{hadoopVersion}"]
+
+    def _getInstalledVersion(self):
+        if not self.jarPath:
+            return None
+        for root, dirs, files in os.walk(self.jarPath):
+            for file in files:
+                if file.startswith("hadoop-client-api-"):
+                    return os.path.splitext(file)[0].replace("hadoop-client-api-", "")
+        return None