marEx 2.0.0__tar.gz → 3.0.2__tar.gz

marex-3.0.2/.github/workflows/README.md

@@ -0,0 +1,205 @@
+# GitHub Actions Workflows for marEx
+
+This directory contains comprehensive GitHub Actions workflows for the marEx package that provide automated testing, code quality checks, performance monitoring, and release management.
+
+## Workflows Overview
+
+### 1. Continuous Integration (`ci.yml`)
+
+**Triggers:**
+- Push to `main` and `advanced_preproc` branches
+- Pull requests to `main` branch
+
+**Features:**
+- **Multi-platform testing**: Ubuntu, Windows, and macOS
+- **Multi-python testing**: Python 3.10, 3.11, and 3.12
+- **Code quality checks**: Black, isort, flake8, bandit, mypy
+- **Comprehensive test suite**: Unit tests, integration tests, error handling tests
+- **Coverage reporting**: Codecov integration and HTML reports
+- **Notebook validation**: Syntax checking and formatting for example notebooks
+- **Build testing**: Package building and installation verification
+- **Dependency caching**: Pip cache for faster builds
+
+**Jobs:**
+1. `code-quality`: Code formatting, linting, and security checks
+2. `test`: Main test suite across multiple OS/Python combinations
+3. `coverage`: Coverage report generation and upload
+4. `test-notebooks`: Notebook syntax and formatting validation
+5. `build-test`: Package building and installation testing
+6. `all-checks-passed`: Final validation of all workflow jobs
+
+### 2. Release Management (`release.yml`)
+
+**Triggers:**
+- Push to version tags (e.g., `v1.0.0`, `v2.1.3`)
+- Manual workflow dispatch
+
+**Features:**
+- **Version validation**: Semantic version tag validation
+- **Full test suite**: Comprehensive testing before release
+- **Multi-platform build testing**: Verify installation across platforms
+- **PyPI publishing**: Automated package publishing using trusted publishing
+- **GitHub release creation**: Automatic release notes and changelog generation
+- **Prerelease support**: Automatic detection of alpha/beta/rc versions
+
+**Jobs:**
+1. `validate-tag`: Version tag validation and prerelease detection
+2. `run-tests`: Full test suite execution
+3. `build`: Source and wheel distribution building
+4. `test-install`: Installation testing across platforms
+5. `publish-pypi`: PyPI package publishing
+6. `create-github-release`: GitHub release creation with changelog
+7. `announce-release`: Post-release notifications (for stable releases)
+
+### 3. Performance Testing (`performance.yml`)
+
+**Triggers:**
+- Push/PR to `main` branch (when performance-related files change)
+- Weekly schedule (Sundays at 2 AM UTC)
+- Manual workflow dispatch
+
+**Features:**
+- **Quick performance tests**: Run on all PRs and pushes
+- **Full performance suite**: Comprehensive benchmarking (main branch only)
+- **Performance regression detection**: Automated performance monitoring
+- **Multiple Python versions**: Testing across supported Python versions
+- **Artifact storage**: Performance results stored for historical analysis
+
+**Jobs:**
+1. `performance-tests`: Quick performance tests for PRs
+2. `full-performance-tests`: Comprehensive benchmarking suite
+
+## Setup Instructions
+
+### Required Secrets
+
+For the workflows to function properly, you need to configure the following secrets in your GitHub repository:
+
+#### 1. CodeCov Integration (Optional)
+- `CODECOV_TOKEN`: Token for uploading coverage reports to Codecov
+  - Go to https://codecov.io/ and connect your repository
+  - Copy the repository token and add it as a secret
+
+#### 2. PyPI Publishing (Required for releases)
+The release workflow uses PyPI's trusted publishing feature, which is more secure than using API tokens.
+
+**Setup Trusted Publishing:**
+1. Go to https://pypi.org/manage/account/publishing/
+2. Add a new pending publisher with these details:
+   - **PyPI Project Name**: `marEx`
+   - **Owner**: `wienkers` (or your GitHub username)
+   - **Repository name**: `marEx`
+   - **Workflow name**: `release.yml`
+   - **Environment name**: `pypi`
+
+**Alternative: Using API Token (less secure)**
+If you prefer to use an API token instead:
+1. Generate a PyPI API token at https://pypi.org/manage/account/token/
+2. Add it as a repository secret named `PYPI_API_TOKEN`
+3. Modify the release workflow to use the token instead of trusted publishing
+
+### Environment Configuration
+
+The release workflow uses a GitHub environment named `pypi` for additional security. To set this up:
+
+1. Go to your repository settings
+2. Navigate to "Environments"
+3. Create a new environment named `pypi`
+4. (Optional) Configure environment protection rules:
+   - Require reviewers for production releases
+   - Restrict to specific branches (e.g., `main`)
+
+## Workflow Customization
+
+### Modifying Test Coverage
+
+To adjust which tests run in different scenarios, modify the test markers in `ci.yml`:
+
+```yaml
+# Skip slow tests in CI
+pytest tests/ -m "not slow"
+
+# Run only integration tests
+pytest tests/ -m "integration"
+
+# Run specific test files
+pytest tests/test_gridded_preprocessing.py -v
+```
+
+### Adding New Platforms
+
+To test on additional platforms, add them to the matrix in `ci.yml`:
+
+```yaml
+strategy:
+  matrix:
+    os: [ubuntu-latest, windows-latest, macos-latest, ubuntu-20.04]
+    python-version: ['3.10', '3.11', '3.12']
+```
+
+### Performance Regression Thresholds
+
+Customize performance regression detection by modifying the performance workflow:
+
+```yaml
+- name: Performance regression check
+  run: |
+    # Add custom logic to compare with baseline performance
+    python scripts/check_performance_regression.py
+```
+
+### Notification Customization
+
+Add custom notifications in the release workflow:
+
+```yaml
+- name: Notify Slack
+  run: |
+    curl -X POST -H 'Content-type: application/json' \
+    --data '{"text":"marEx v${{ needs.validate-tag.outputs.version }} released!"}' \
+    ${{ secrets.SLACK_WEBHOOK_URL }}
+```
+
+## Workflow Dependencies
+
+The workflows require these system dependencies:
+
+**Ubuntu:**
+```bash
+libhdf5-dev libnetcdf-dev libproj-dev proj-data proj-bin libgeos-dev
+```
+
+**macOS:**
+```bash
+hdf5 netcdf proj geos
+```
+
+**Windows:**
+Dependencies are handled through pip packages.
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Build failures on Windows**: Ensure all dependencies are properly specified in `pyproject.toml`
+2. **Coverage upload failures**: Check that `CODECOV_TOKEN` is correctly configured
+3. **Release workflow failures**: Verify that PyPI trusted publishing is properly configured
+4. **Performance test timeouts**: Adjust timeout values in the workflow if needed
+
+### Debugging Tips
+
+1. **Check workflow logs**: Go to Actions tab in your repository to view detailed logs
+2. **Local testing**: Run tests locally with the same commands used in CI
+3. **Matrix debugging**: Use `strategy.fail-fast: false` to see all platform failures
+4. **Artifact inspection**: Download artifacts from failed runs to analyze results
+
+## Contributing
+
+When contributing to marEx, please ensure that:
+
+1. Your changes pass all CI checks
+2. New features include appropriate tests
+3. Documentation is updated if needed
+4. Performance-critical changes include performance tests
+
+The workflows will automatically run on your pull requests and provide feedback on code quality, test coverage, and performance impact.

marex-3.0.2/.github/workflows/ci.yml

@@ -0,0 +1,328 @@
+name: Continuous Integration
+
+on:
+  push:
+    branches: [ main, advanced_preproc ]
+  pull_request:
+    branches: [ main ]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  code-quality:
+    name: Code Quality Checks
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+        cache: 'pip'
+        cache-dependency-path: 'pyproject.toml'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e ".[dev]"
+
+    - name: Check code formatting with Black
+      run: |
+        black --check --diff marEx/
+
+    - name: Check import sorting with isort
+      run: |
+        isort --check-only --diff marEx/
+
+    - name: Lint with flake8
+      run: |
+        flake8 marEx/
+
+    - name: Security check with bandit
+      run: |
+        bandit -r marEx/ -f json -o bandit-report.json || true
+
+    - name: Type checking with mypy
+      run: |
+        mypy marEx/ || true
+
+    - name: Upload bandit report
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: bandit-report
+        path: bandit-report.json
+        retention-days: 7
+
+  test:
+    name: Test Suite
+    runs-on: ${{ matrix.os }}
+    needs: code-quality
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ['3.10', '3.11', '3.12']
+        exclude:
+          # Reduce matrix size for faster CI
+          - os: windows-latest
+            python-version: '3.10'
+          - os: macos-latest
+            python-version: '3.10'
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+        cache: 'pip'
+        cache-dependency-path: 'pyproject.toml'
+
+    - name: Install system dependencies (Ubuntu)
+      if: matrix.os == 'ubuntu-latest'
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libhdf5-dev libnetcdf-dev libproj-dev proj-data proj-bin libgeos-dev
+
+    - name: Install system dependencies (macOS)
+      if: matrix.os == 'macos-latest'
+      run: |
+        brew install hdf5 netcdf proj geos
+
+    - name: Install system dependencies (Windows)
+      if: matrix.os == 'windows-latest'
+      run: |
+        # Windows dependencies are handled by conda-forge packages
+        echo "Windows dependencies handled by pip packages"
+
+    - name: Install Python dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e ".[dev,full]"
+
+    - name: Configure Dask for CI
+      run: |
+        mkdir -p ~/.dask
+        echo "distributed:" > ~/.dask/config.yaml
+        echo "  worker:" >> ~/.dask/config.yaml
+        echo "    memory:" >> ~/.dask/config.yaml
+        echo "      target: 0.8" >> ~/.dask/config.yaml
+        echo "      spill: 0.9" >> ~/.dask/config.yaml
+        echo "      pause: 0.95" >> ~/.dask/config.yaml
+        echo "      terminate: 0.98" >> ~/.dask/config.yaml
+        echo "  admin:" >> ~/.dask/config.yaml
+        echo "    log-format: '%(name)s - %(levelname)s - %(message)s'" >> ~/.dask/config.yaml
+        echo "array:" >> ~/.dask/config.yaml
+        echo "  chunk-size: 64MiB" >> ~/.dask/config.yaml
+
+    - name: List installed packages
+      run: pip list
+
+    - name: Run unit tests
+      run: |
+        pytest tests/test_detect_helpers.py tests/test_track_helpers.py -v --tb=short
+
+    - name: Run error handling tests
+      run: |
+        pytest tests/test_error_handling.py -v --tb=short
+
+    - name: Run exception tests
+      run: |
+        pytest tests/test_exceptions.py -v --tb=short
+
+    - name: Run logging system tests
+      run: |
+        pytest tests/test_logging_system.py -v --tb=short
+
+    - name: Run preprocessing tests (gridded)
+      run: |
+        pytest tests/test_gridded_preprocessing.py -v --tb=short
+
+    - name: Run preprocessing tests (unstructured)
+      run: |
+        pytest tests/test_unstructured_preprocessing.py -v --tb=short
+
+    - name: Run tracking tests (gridded)
+      run: |
+        pytest tests/test_gridded_tracking.py -v --tb=short
+
+    - name: Run tracking tests (unstructured)
+      run: |
+        pytest tests/test_unstructured_tracking.py -v --tb=short
+
+    - name: Run plotting tests
+      run: |
+        pytest tests/test_plotx.py -v --tb=short
+
+    - name: Run integration tests
+      run: |
+        pytest tests/test_integration.py -v --tb=short -m "not slow"
+
+  coverage:
+    name: Coverage Report
+    runs-on: ubuntu-latest
+    needs: test
+    if: always()
+    timeout-minutes: 60
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+        cache: 'pip'
+        cache-dependency-path: 'pyproject.toml'
+
+    - name: Install system dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libhdf5-dev libnetcdf-dev libproj-dev proj-data proj-bin libgeos-dev
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e ".[dev,full]"
+
+    - name: Configure Dask for Coverage (Conservative)
+      run: |
+        mkdir -p ~/.dask
+        echo "distributed:" > ~/.dask/config.yaml
+        echo "  worker:" >> ~/.dask/config.yaml
+        echo "    memory:" >> ~/.dask/config.yaml
+        echo "      target: 0.5" >> ~/.dask/config.yaml
+        echo "      spill: 0.6" >> ~/.dask/config.yaml
+        echo "      pause: 0.7" >> ~/.dask/config.yaml
+        echo "      terminate: 0.8" >> ~/.dask/config.yaml
+        echo "      recent-to-old-time: 15s" >> ~/.dask/config.yaml
+        echo "    daemon: false" >> ~/.dask/config.yaml
+        echo "  scheduler:" >> ~/.dask/config.yaml
+        echo "    allowed-failures: 100" >> ~/.dask/config.yaml
+        echo "    work-stealing: false" >> ~/.dask/config.yaml
+        echo "    worker-ttl: 600s" >> ~/.dask/config.yaml
+        echo "  comm:" >> ~/.dask/config.yaml
+        echo "    timeouts:" >> ~/.dask/config.yaml
+        echo "      connect: 600s" >> ~/.dask/config.yaml
+        echo "      tcp: 600s" >> ~/.dask/config.yaml
+        echo "    retry:" >> ~/.dask/config.yaml
+        echo "      count: 20" >> ~/.dask/config.yaml
+        echo "      delay:" >> ~/.dask/config.yaml
+        echo "        min: 3s" >> ~/.dask/config.yaml
+        echo "        max: 60s" >> ~/.dask/config.yaml
+        echo "  admin:" >> ~/.dask/config.yaml
+        echo "    log-format: '%(name)s - %(levelname)s - %(message)s'" >> ~/.dask/config.yaml
+        echo "array:" >> ~/.dask/config.yaml
+        echo "  chunk-size: 24MiB" >> ~/.dask/config.yaml
+        echo "  slicing:" >> ~/.dask/config.yaml
+        echo "    split-large-chunks: false" >> ~/.dask/config.yaml
+
+    - name: Set environment for conservative execution
+      run: |
+        echo "DASK_NUM_WORKERS=1" >> $GITHUB_ENV
+        echo "OMP_NUM_THREADS=1" >> $GITHUB_ENV
+        echo "MKL_NUM_THREADS=1" >> $GITHUB_ENV
+        echo "NUMBA_NUM_THREADS=1" >> $GITHUB_ENV
+        echo "PYTHONHASHSEED=42" >> $GITHUB_ENV
+        echo "PYTEST_COVERAGE=true" >> $GITHUB_ENV
+
+    - name: Generate coverage report
+      timeout-minutes: 45
+      run: |
+        # Run with parallel mode and multiprocessing support
+        coverage run --parallel-mode --rcfile=.coveragerc -m pytest tests/ --tb=short -m "not nocov" -x --maxfail=3
+        # Combine all coverage data files from subprocesses
+        coverage combine
+        coverage report -m
+        coverage html
+        coverage xml
+
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v4
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        files: ./coverage.xml
+        fail_ci_if_error: false
+        verbose: true
+
+    - name: Upload coverage HTML report
+      uses: actions/upload-artifact@v4
+      with:
+        name: coverage-report
+        path: htmlcov/
+        retention-days: 90
+
+
+  build-test:
+    name: Build and Test Package
+    runs-on: ubuntu-latest
+    needs: test
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.11'
+        cache: 'pip'
+        cache-dependency-path: 'pyproject.toml'
+
+    - name: Install build dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install build twine
+
+    - name: Build package
+      run: |
+        python -m build
+
+    - name: Check package integrity
+      run: |
+        twine check dist/*
+
+    - name: Test package installation
+      shell: bash
+      run: |
+        pip install dist/*.whl
+        python -c "import marEx; print(f'marEx version: {marEx.__version__}')"
+
+    - name: Upload build artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: dist-packages
+        path: dist/
+        retention-days: 7
+
+  all-checks-passed:
+    name: All Checks Passed
+    runs-on: ubuntu-latest
+    needs: [code-quality, test, coverage, build-test]
+    if: always()
+
+    steps:
+    - name: Check all jobs status
+      run: |
+        if [[ "${{ needs.code-quality.result }}" == "success" &&
+              "${{ needs.test.result }}" == "success" &&
+              "${{ needs.coverage.result }}" == "success" &&
+              "${{ needs.build-test.result }}" == "success" ]]; then
+          echo "All checks passed!"
+        else
+          echo "Some checks failed:"
+          echo "Code Quality: ${{ needs.code-quality.result }}"
+          echo "Tests: ${{ needs.test.result }}"
+          echo "Coverage: ${{ needs.coverage.result }}"
+          echo "Build Test: ${{ needs.build-test.result }}"
+          exit 1
+        fi