marEx 3.1__tar.gz → 4.0__tar.gz

{marex-3.1 → marex-4.0}/.github/workflows/README.md

@@ -51,30 +51,10 @@ This directory contains comprehensive GitHub Actions workflows for the marEx pac
 6. `create-github-release`: GitHub release creation with changelog
 7. `announce-release`: Post-release notifications (for stable releases)
 
-### 3. Performance Testing (`performance.yml`)
-
-**Triggers:**
-- Push/PR to `main` branch (when performance-related files change)
-- Weekly schedule (Sundays at 2 AM UTC)
-- Manual workflow dispatch
-
-**Features:**
-- **Quick performance tests**: Run on all PRs and pushes
-- **Full performance suite**: Comprehensive benchmarking (main branch only)
-- **Performance regression detection**: Automated performance monitoring
-- **Multiple Python versions**: Testing across supported Python versions
-- **Artifact storage**: Performance results stored for historical analysis
-
-**Jobs:**
-1. `performance-tests`: Quick performance tests for PRs
-2. `full-performance-tests`: Comprehensive benchmarking suite
-
 ## Setup Instructions
 
 ### Required Secrets
 
-For the workflows to function properly, you need to configure the following secrets in your GitHub repository:
-
 #### 1. CodeCov Integration (Optional)
 - `CODECOV_TOKEN`: Token for uploading coverage reports to Codecov
   - Go to https://codecov.io/ and connect your repository
@@ -137,29 +117,6 @@ strategy:
     python-version: ['3.10', '3.11', '3.12']
 ```
 
-### Performance Regression Thresholds
-
-Customize performance regression detection by modifying the performance workflow:
-
-```yaml
-- name: Performance regression check
-  run: |
-    # Add custom logic to compare with baseline performance
-    python scripts/check_performance_regression.py
-```
-
-### Notification Customization
-
-Add custom notifications in the release workflow:
-
-```yaml
-- name: Notify Slack
-  run: |
-    curl -X POST -H 'Content-type: application/json' \
-    --data '{"text":"marEx v${{ needs.validate-tag.outputs.version }} released!"}' \
-    ${{ secrets.SLACK_WEBHOOK_URL }}
-```
-
 ## Workflow Dependencies
 
 The workflows require these system dependencies:
@@ -201,5 +158,3 @@ When contributing to marEx, please ensure that:
 2. New features include appropriate tests
 3. Documentation is updated if needed
 4. Performance-critical changes include performance tests
-
-The workflows will automatically run on your pull requests and provide feedback on code quality, test coverage, and performance impact.

{marex-3.1 → marex-4.0}/.github/workflows/ci.yml

@@ -1,10 +1,11 @@
-name: Continuous Integration
+name: CI
 
 on:
   push:
     branches: [ main, advanced_preproc ]
   pull_request:
     branches: [ main ]
+  workflow_call:
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -128,50 +129,50 @@ jobs:
 
     - name: Run unit tests
       run: |
-        pytest tests/test_detect_helpers.py tests/test_track_helpers.py -v --tb=short
+        pytest tests/test_detect_helpers.py tests/test_track_helpers.py -n 2 -v --tb=short
 
     - name: Run error handling tests
       run: |
-        pytest tests/test_error_handling.py -v --tb=short
+        pytest tests/test_error_handling.py -n 2 -v --tb=short
 
     - name: Run exception tests
       run: |
-        pytest tests/test_exceptions.py -v --tb=short
+        pytest tests/test_exceptions.py -n 2 -v --tb=short
 
     - name: Run logging system tests
       run: |
-        pytest tests/test_logging_system.py -v --tb=short
+        pytest tests/test_logging_system.py -n 2 -v --tb=short
 
     - name: Run preprocessing tests (gridded)
       run: |
-        pytest tests/test_gridded_preprocessing.py -v --tb=short
+        pytest tests/test_gridded_preprocessing.py -n 2 -v --tb=short
 
     - name: Run preprocessing tests (unstructured)
       run: |
-        pytest tests/test_unstructured_preprocessing.py -v --tb=short
+        pytest tests/test_unstructured_preprocessing.py -n 2 -v --tb=short
 
     - name: Run tracking tests (gridded)
       run: |
-        pytest tests/test_gridded_tracking.py -v --tb=short
+        pytest tests/test_gridded_tracking.py -n 2 -v --tb=short
 
     - name: Run tracking tests (unstructured)
       run: |
-        pytest tests/test_unstructured_tracking.py -v --tb=short
+        pytest tests/test_unstructured_tracking.py -n 2 -v --tb=short
 
     - name: Run plotting tests
       run: |
-        pytest tests/test_plotx.py -v --tb=short
+        pytest tests/test_plotx.py -n 2 -v --tb=short
 
     - name: Run integration tests
       run: |
-        pytest tests/test_integration.py -v --tb=short -m "not slow"
+        pytest tests/test_integration.py -n 2 -v --tb=short -m "not slow"
 
   coverage:
     name: Coverage Report
     runs-on: ubuntu-latest
     needs: test
     if: always()
-    timeout-minutes: 60
+    timeout-minutes: 150
 
     steps:
     - name: Checkout code
@@ -236,15 +237,10 @@ jobs:
         echo "PYTEST_COVERAGE=true" >> $GITHUB_ENV
 
     - name: Generate coverage report
-      timeout-minutes: 45
+      timeout-minutes: 120
       run: |
-        # Run with parallel mode and multiprocessing support
-        coverage run --parallel-mode --rcfile=.coveragerc -m pytest tests/ --tb=short -m "not nocov" -x --maxfail=3
-        # Combine all coverage data files from subprocesses
-        coverage combine
-        coverage report -m
-        coverage html
-        coverage xml
+        # Run tests in parallel with coverage collection
+        pytest tests/ -n auto --cov=marEx --cov-report=term --cov-report=html --cov-report=xml --cov-config=.coveragerc --tb=short -m "not nocov" -x --maxfail=3
 
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@v4

{marex-3.1 → marex-4.0}/.github/workflows/release.yml

@@ -276,10 +276,26 @@ jobs:
         generate_release_notes: true
         make_latest: ${{ needs.validate-tag.outputs.is_prerelease == 'false' }}
 
+  trigger-docs:
+    name: Trigger ReadTheDocs Build
+    runs-on: ubuntu-latest
+    needs: [validate-tag, publish-pypi]
+
+    steps:
+    - name: Trigger ReadTheDocs build
+      run: |
+        if [ -n "${{ secrets.READTHEDOCS_WEBHOOK_URL }}" ]; then
+          curl -X POST -d "branches=main" -d "token=${{ secrets.READTHEDOCS_WEBHOOK_TOKEN }}" \
+            ${{ secrets.READTHEDOCS_WEBHOOK_URL }}
+          echo "✅ ReadTheDocs build triggered for v${{ needs.validate-tag.outputs.version }}"
+        else
+          echo "ℹ️  No ReadTheDocs webhook configured. Docs will build automatically on next push."
+        fi
+
   announce-release:
     name: Post-Release Notifications
     runs-on: ubuntu-latest
-    needs: [validate-tag, create-github-release]
+    needs: [validate-tag, create-github-release, trigger-docs]
     if: needs.validate-tag.outputs.is_prerelease == 'false'  # Only for stable releases
 
     steps:
@@ -288,6 +304,7 @@ jobs:
         echo "🎉 marEx v${{ needs.validate-tag.outputs.version }} has been successfully released!"
         echo "📦 PyPI: https://pypi.org/project/marEx/${{ needs.validate-tag.outputs.version }}/"
         echo "📋 GitHub: https://github.com/${{ github.repository }}/releases/tag/${{ github.ref_name }}"
+        echo "📚 Docs: https://marex.readthedocs.io/"
 
         # You can add additional notification steps here, such as:
         # - Slack notifications

marex-4.0/PKG-INFO

@@ -0,0 +1,297 @@
+Metadata-Version: 2.4
+Name: marEx
+Version: 4.0
+Summary: Marine Extremes Detection and Tracking
+Author-email: Aaron Wienkers <aaron@wienkers.com>
+Project-URL: Homepage, https://github.com/wienkers/marEx
+Project-URL: Bug Tracker, https://github.com/wienkers/marEx/issues
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
+Classifier: Topic :: Scientific/Engineering :: Physics
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: xarray[complete]
+Requires-Dist: dask[complete]==2025.3.0
+Requires-Dist: netcdf4
+Requires-Dist: scipy
+Requires-Dist: scikit-image
+Requires-Dist: dask_image
+Requires-Dist: pandas
+Requires-Dist: numba
+Requires-Dist: pillow
+Requires-Dist: xhistogram
+Requires-Dist: flox>=0.10.1
+Requires-Dist: matplotlib
+Requires-Dist: cartopy
+Provides-Extra: minimal
+Provides-Extra: performance
+Requires-Dist: jax; extra == "performance"
+Requires-Dist: jaxlib; extra == "performance"
+Provides-Extra: hpc
+Requires-Dist: dask_jobqueue; extra == "hpc"
+Requires-Dist: psutil; extra == "hpc"
+Provides-Extra: plotting
+Requires-Dist: seaborn>=0.11.0; extra == "plotting"
+Requires-Dist: cmocean; extra == "plotting"
+Requires-Dist: cartopy; extra == "plotting"
+Requires-Dist: matplotlib>=3.5.0; extra == "plotting"
+Requires-Dist: ffmpeg; extra == "plotting"
+Provides-Extra: full
+Requires-Dist: jax; extra == "full"
+Requires-Dist: jaxlib; extra == "full"
+Requires-Dist: dask_jobqueue; extra == "full"
+Requires-Dist: seaborn>=0.11.0; extra == "full"
+Requires-Dist: cmocean; extra == "full"
+Requires-Dist: ffmpeg; extra == "full"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-xdist; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: flake8>=4.0.0; extra == "dev"
+Requires-Dist: flake8-pyproject; extra == "dev"
+Requires-Dist: isort>=5.0.0; extra == "dev"
+Requires-Dist: mypy>=0.991; extra == "dev"
+Requires-Dist: pre-commit>=2.20.0; extra == "dev"
+Requires-Dist: coverage[toml]>=6.0.0; extra == "dev"
+Requires-Dist: nbqa; extra == "dev"
+Requires-Dist: bandit[toml]>=1.7.0; extra == "dev"
+Requires-Dist: psutil; extra == "dev"
+Requires-Dist: sphinx>=5.0.0; extra == "dev"
+Requires-Dist: sphinx-rtd-theme; extra == "dev"
+Requires-Dist: myst-parser; extra == "dev"
+Requires-Dist: jupyter; extra == "dev"
+Requires-Dist: ipykernel; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=5.0.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme; extra == "docs"
+Requires-Dist: myst-parser; extra == "docs"
+Requires-Dist: sphinx-autodoc-typehints; extra == "docs"
+Requires-Dist: nbsphinx; extra == "docs"
+Requires-Dist: pandoc; extra == "docs"
+Requires-Dist: sphinxcontrib-video; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Requires-Dist: pytest-xdist; extra == "test"
+Requires-Dist: coverage[toml]>=6.0.0; extra == "test"
+Requires-Dist: psutil; extra == "test"
+
+![logo](https://github.com/user-attachments/assets/df99a84b-c88f-447d-a362-ac538d9b924b)
+
+[![CI](https://github.com/wienkers/marEx/actions/workflows/ci.yml/badge.svg)](https://github.com/wienkers/marEx/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/wienkers/marEx/branch/main/graph/badge.svg)](https://codecov.io/gh/wienkers/marEx)
+[![PyPI version](https://badge.fury.io/py/marEx.svg)](https://badge.fury.io/py/marEx)
+[![Documentation Status](https://readthedocs.org/projects/marex/badge/?version=latest)](https://marex.readthedocs.io/en/latest/)
+[![PyPI Downloads](https://static.pepy.tech/badge/marex)](https://pepy.tech/projects/marex)
+[![DOI](https://zenodo.org/badge/945834123.svg)](https://doi.org/10.5281/zenodo.16922881)
+
+# Marine Extremes Detection and Tracking
+
+**Efficient & scalable marine extremes detection, identification, & tracking for exascale climate data.**
+
+marEx is a high-performance Python framework for identifying and tracking extreme oceanographic events (such as Marine Heatwaves or Acidity Extremes) in massive climate datasets. Built on advanced statistical methods and distributed computing, it processes decades of daily-resolution global ocean data with unprecedented efficiency and scalability.
+
+---
+
+## Key Features
+
+- **⚡ Extreme Performance**: Process 100+ years of high-resolution daily global data in minutes
+- **🌐 Universal Grid Support**: Native support for both regular (lat/lon) grids and unstructured ocean models
+- **📈 Advanced Event Tracking**: Handles coherent object splitting, merging, and evolution
+- **📊 Multiple Detection Methods**: Scientifically rigorous algorithms for robust extreme event identification
+- **☁️ Cloud-Native Scaling**: Identical codebase scales from laptop to supercomputer using up to 1024+ cores
+- **🧠 Memory Efficient**: Intelligent chunking and lazy evaluation for datasets larger than memory
+
+
+---
+
+https://github.com/user-attachments/assets/501537ff-5adb-4e13-ba08-6a333bac2a02
+
+![marEx_front](https://github.com/user-attachments/assets/939fceee-8990-46fb-b3f8-30e803b6c802)
+
+---
+
+## Quick Start
+
+```python
+import xarray as xr
+import marEx
+
+# Load sea surface temperature data
+sst = xr.open_dataset('sst_data.nc', chunks={'time': 30}).sst
+
+# Identify extreme events
+extreme_events_ds = marEx.preprocess_data(
+    sst,
+    threshold_percentile=95,
+    method_anomaly='shifting_baseline',
+    method_extreme='hobday_extreme'
+)
+
+# Track events through time
+events_ds = marEx.tracker(
+    extreme_events_ds.extreme_events,
+    extreme_events_ds.mask,
+    R_fill=8,
+    area_filter_absolute=100,
+    allow_merging=True
+).run()
+
+# Visualise results
+fig, ax, im = (events_ds.ID_field > 0).mean("time").plotX.single_plot(
+    marEx.PlotConfig(var_units="MHW Frequency", cmap="hot_r", cperc=[0, 96])
+)
+```
+
+---
+
+## Installation
+
+```bash
+pip install marEx[full,hpc]
+```
+
+For detailed installation instructions, including HPC environments and optional dependencies, see the **[Installation Guide](https://marex.readthedocs.io/en/latest/installation.html)**.
+
+---
+
+## Core Workflow
+
+marEx follows a three-stage pipeline:
+
+```
+┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
+│  1. Detect      │  →   │  2. Track       │  →   │  3. Visualise   │
+│    Extremes     │      │    Events       │      │     & Analyse   │
+└─────────────────┘      └─────────────────┘      └─────────────────┘
+        ↓                        ↓                        ↓
+preprocess_data()           tracker()                  plotX()
+        ↓                        ↓                        ↓
+Binary extreme map        Tracked objects          Maps, animations,
+                            with unique IDs           & statistics
+```
+
+**Learn more**: [Core Concepts](https://marex.readthedocs.io/en/latest/concepts.html) | [User Guide](https://marex.readthedocs.io/en/latest/user_guide.html)
+
+---
+
+## Documentation
+
+For detailed guides, tutorials, and API reference, visit the full documentation:
+
+### 📚 **[marEx Documentation on ReadTheDocs](https://marex.readthedocs.io/)**
+
+**Quick Links:**
+- **[Quickstart Guide](https://marex.readthedocs.io/en/latest/quickstart.html)** - Get started in 5 minutes
+- **[Why marEx?](https://marex.readthedocs.io/en/latest/why_marex.html)** - Learn about the unique capabilities
+- **[Core Concepts](https://marex.readthedocs.io/en/latest/concepts.html)** - Understanding marEx's design and workflow
+- **[User Guide](https://marex.readthedocs.io/en/latest/user_guide.html)** - Comprehensive usage guide with method selection, parameter tuning, and performance optimisation
+- **[API Reference](https://marex.readthedocs.io/en/latest/api.html)** - Complete function documentation
+- **[Examples](https://marex.readthedocs.io/en/latest/examples.html)** - Jupyter notebooks for gridded, regional, and unstructured data
+- **[Troubleshooting](https://marex.readthedocs.io/en/latest/troubleshooting.html)** - Common issues and solutions
+
+---
+
+## Examples
+
+Explore complete workflows in the [example notebooks](https://github.com/wienkers/marEx/tree/main/examples):
+
+- **Gridded Data**: Standard analysis for regular lat/lon grids (satellite data, CMIP6 models)
+- **Regional Data**: Regional analysis with boundary handling (EURO-CORDEX)
+- **Unstructured Data**: Analysis for irregular meshes (FESOM, ICON-O, MPAS-Ocean)
+
+Each example demonstrates the full pipeline from preprocessing to visualisation.
+
+---
+
+## Key Capabilities
+
+### Detection Methods
+
+marEx provides multiple scientifically rigorous methods for anomaly calculation and extreme identification:
+
+**Anomaly Detection:**
+- **Shifting Baseline**: Rolling climatology (most accurate, research standard)
+- **Detrend Fixed Baseline**: Polynomial detrending + fixed climatology (preserves full time series)
+- **Fixed Baseline**: Simple daily climatology (trend-inclusive)
+- **Harmonic Detrending**: Fast polynomial + harmonic model (efficient screening)
+
+**Extreme Identification:**
+- **Hobday Method**: Day-of-year specific thresholds with spatial window extension (literature standard, Hobday et al. 2016)
+- **Global Method**: Single threshold across time (fast, exploratory analysis)
+
+**[→ Learn more about method selection](https://marex.readthedocs.io/en/latest/user_guide.html#method-selection-guide)**
+
+### Advanced Tracking
+
+- **Morphological Operations**: Fill spatial gaps and smooth event boundaries
+- **Temporal Gap Filling**: Maintain event continuity across short interruptions
+- **Merge/Split Handling**: Track event genealogy with improved nearest-neighbor partitioning
+- **Area Filtering**: Remove spurious small events with percentile or absolute thresholds
+
+**[→ Explore tracking algorithms](https://marex.readthedocs.io/en/latest/modules/track.html)**
+
+### Performance & Scalability
+
+- **Dask-First Architecture**: Parallel computation with automatic memory management
+- **JAX Acceleration**: Optional GPU/TPU support for 10-50× speedup
+- **HPC Integration**: SLURM cluster support for supercomputing environments
+- **Memory Optimisation**: Process datasets 100-1000× larger than available RAM
+
+**[→ Performance tuning guide](https://marex.readthedocs.io/en/latest/user_guide.html#performance-optimisation)**
+
+---
+
+## Getting Help
+
+### Support Channels
+
+- **[Documentation](https://marex.readthedocs.io/)** - Detailed guides and API reference
+- **[GitHub Issues](https://github.com/wienkers/marEx/issues)** - Bug reports and feature requests
+- **[GitHub Discussions](https://github.com/wienkers/marEx/discussions)** - Questions, ideas, and community support
+- **[Example Notebooks](https://github.com/wienkers/marEx/tree/main/examples)** - Complete workflow demonstrations
+
+### Reporting Issues
+
+When reporting issues, please include:
+- marEx version (`marEx.__version__`)
+- Python version and operating system
+- Dependency status (`marEx.print_dependency_status()`)
+- Minimal reproducible example
+- Full error traceback
+
+---
+
+## Citation
+
+When using marEx in publications, please cite:
+
+- **marEx package**: DOI [10.5281/zenodo.16922881](https://doi.org/10.5281/zenodo.16922881)
+- **Hobday et al. (2016)**: "A hierarchical approach to defining marine heatwaves." *Progress in Oceanography* 141, 227-238. DOI [10.1016/j.pocean.2015.12.014](https://doi.org/10.1016/j.pocean.2015.12.014)
+
+---
+
+## Funding
+
+This project has received funding through:
+
+* The [EERIE](https://eerie-project.eu) (European Eddy-Rich ESMs) Project
+* The European Union's Horizon Europe research and innovation programme under Grant Agreement No. 101081383
+* The Swiss State Secretariat for Education, Research and Innovation (SERI) under contract #22.00366
+
+---
+
+## Contact
+
+For questions, comments, or collaboration opportunities, please contact [Aaron Wienkers](mailto:aaron.wienkers@gmail.com).