arraybridge 0.2.0__tar.gz → 0.2.9__tar.gz

arraybridge-0.2.9/.github/badges/coverage.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="114" height="20" role="img" aria-label="coverage: 77.70%"><title>coverage: 77.70%</title><linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient><clipPath id="r"><rect width="114" height="20" rx="3" fill="#fff"/></clipPath><g clip-path="url(#r)"><rect width="61" height="20" fill="#555"/><rect x="61" width="53" height="20" fill="#97ca00"/><rect width="114" height="20" fill="url(#s)"/></g><g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" text-rendering="geometricPrecision" font-size="110"><text aria-hidden="true" x="315" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="510">coverage</text><text x="315" y="140" transform="scale(.1)" fill="#fff" textLength="510">coverage</text><text aria-hidden="true" x="865" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="430">77.70%</text><text x="865" y="140" transform="scale(.1)" fill="#fff" textLength="430">77.70%</text></g></svg>

arraybridge-0.2.9/.github/workflows/ci.yml

@@ -0,0 +1,158 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+    branches: [main, master]
+  workflow_dispatch:
+
+jobs:
+  # Test across Python versions with CPU-compatible frameworks (no torch)
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install base dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run tests with coverage
+        run: |
+          pytest --cov=arraybridge --cov-report=xml --cov-report=html --cov-report=term-missing -v
+
+      - name: Upload coverage to Codecov
+        if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.12'
+        uses: codecov/codecov-action@v3
+        with:
+          file: ./coverage.xml
+          fail_ci_if_error: false
+
+  # GPU tests - includes framework testing
+  # Note: GitHub Actions ubuntu-latest doesn't have physical GPU,
+  # but tests will run the "unavailable GPU" code paths and mock GPU tests
+  gpu-test:
+    runs-on: ubuntu-latest
+    continue-on-error: true  # Don't block PR merges if GPU not available
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install base dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Install GPU frameworks (will use CPU versions in CI)
+        run: |
+          # PyTorch - CPU version will be installed in CI (no GPU available)
+          pip install torch torchvision torchaudio 2>&1 || echo "PyTorch install attempted"
+          
+          # JAX - CPU version
+          pip install jax jaxlib 2>&1 || echo "JAX install skipped (optional)"
+          
+          # CuPy - will fail without CUDA, that's ok
+          pip install cupy-cuda12x 2>&1 || echo "CuPy skipped (requires actual CUDA)"
+
+      - name: Check framework availability
+        run: |
+          python -c "
+          print('=== Framework Availability Check ===')
+          try:
+              import torch
+              print(f'✓ PyTorch available')
+              print(f'  CUDA available: {torch.cuda.is_available()}')
+              print(f'  (This is normal - GitHub Actions has no physical GPU)')
+          except ImportError:
+              print('✗ PyTorch not available')
+          
+          try:
+              import jax
+              print(f'✓ JAX available')
+          except ImportError:
+              print('✗ JAX not available')
+          
+          try:
+              import cupy
+              print(f'✓ CuPy available')
+          except ImportError:
+              print('✗ CuPy not available (normal - requires CUDA)')
+          " || true
+
+      - name: Run GPU cleanup tests
+        run: |
+          # Tests include:
+          # 1. Framework unavailable tests (always run)
+          # 2. GPU unavailable fallback paths (will run in CI)
+          # 3. Mocked GPU tests (test cleanup code with mocked GPU state)
+          pytest -v tests/test_gpu_cleanup.py \
+            --cov=arraybridge \
+            --cov-report=term-missing \
+            --cov-report=html \
+            --cov-report=xml \
+            -ra
+
+      - name: Upload GPU test coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          file: ./coverage.xml
+          flags: gpu-tests
+          fail_ci_if_error: false
+
+      - name: Upload HTML coverage report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gpu-test-coverage-report
+          path: htmlcov/
+
+  # Code quality checks
+  code-quality:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install ruff black mypy
+
+      - name: Run ruff (linting)
+        run: |
+          ruff check src/ --output-format=github
+
+      - name: Run black (formatting check)
+        run: |
+          black --check src/
+
+      - name: Run mypy (type checking)
+        continue-on-error: true  # Scientific code often has complex types
+        run: |
+          mypy src/ --ignore-missing-imports

arraybridge-0.2.9/.github/workflows/coverage-pages.yml

@@ -0,0 +1,115 @@
+name: Tests, Coverage and GitHub Pages
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: write  # Needed for badge commit
+  pages: write
+  id-token: write
+
+concurrency:
+  group: "pages"
+  cancel-in-progress: true
+
+jobs:
+  test-and-deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+          pip install genbadge[coverage]
+
+      - name: Install PyTorch (CPU-only) for tests
+        run: |
+          pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+      - name: Run tests with coverage
+        continue-on-error: true  # Continue workflow even if tests fail
+        run: |
+          mkdir -p site
+          python -m pytest --cov=arraybridge \
+                --cov-report=xml \
+                --cov-report=html:site/coverage \
+                tests/
+
+      - name: Generate coverage badge
+        run: |
+          mkdir -p .github/badges
+          genbadge coverage -i coverage.xml -o .github/badges/coverage.svg -n "coverage"
+
+      - name: Create index.html and README
+        run: |
+          # Create index.html for redirection
+          cat > site/index.html << 'EOF'
+          <!DOCTYPE html>
+          <html>
+            <head>
+              <meta http-equiv="refresh" content="0; url=./coverage/">
+              <title>Redirecting to coverage report...</title>
+            </head>
+            <body>
+              <p>Redirecting to coverage report... <a href="./coverage/">Click here if not redirected</a></p>
+            </body>
+          </html>
+          EOF
+
+          # Create README.md in the site directory
+          echo "# arraybridge Code Coverage Reports" > site/README.md
+          echo "" >> site/README.md
+          echo "This site contains the code coverage reports for the [arraybridge](https://github.com/trissim/arraybridge) project." >> site/README.md
+          echo "" >> site/README.md
+          echo "## Navigation" >> site/README.md
+          echo "" >> site/README.md
+          echo "- [Coverage Report](./coverage/): View the HTML coverage report" >> site/README.md
+          echo "" >> site/README.md
+          echo "## About" >> site/README.md
+          echo "" >> site/README.md
+          echo "These reports are automatically generated by GitHub Actions whenever changes are pushed to the main branch." >> site/README.md
+          echo "They show the percentage of code that is covered by automated tests." >> site/README.md
+          echo "" >> site/README.md
+          echo "Last updated: $(date)" >> site/README.md
+
+      - name: Commit and push if coverage badge changed
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        run: |
+          git config --local user.email "github-actions[bot]@users.noreply.github.com"
+          git config --local user.name "github-actions[bot]"
+          git add .github/badges/coverage.svg -f
+          git commit -m "chore: update coverage badge" || exit 0
+          git push
+
+      - name: Check GitHub Pages status
+        run: |
+          echo "⚠️ IMPORTANT: Make sure GitHub Pages is enabled in your repository settings!"
+          echo "Go to https://github.com/trissim/arraybridge/settings/pages"
+          echo "Set 'Source' to 'GitHub Actions' to enable GitHub Pages deployment."
+
+      - name: Setup Pages
+        uses: actions/configure-pages@v4
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: 'site'
+          retention-days: 1
+
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4
+        timeout-minutes: 10

arraybridge-0.2.9/.github/workflows/gpu-tests.yml

@@ -0,0 +1,79 @@
+name: GPU Tests (Manual - Comprehensive GPU Testing)
+
+on:
+  workflow_dispatch:  # Manual trigger only
+
+jobs:
+  gpu-test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install base dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Install CPU-available frameworks
+        run: |
+          # Install CPU versions of frameworks for testing
+          # (Real GPU tests would need actual CUDA infrastructure)
+          pip install torch --index-url https://download.pytorch.org/whl/cpu
+          pip install jax jaxlib
+          echo "Installed PyTorch (CPU) and JAX for testing"
+
+      - name: Check framework availability
+        run: |
+          python -c "
+          import sys
+          frameworks = ['numpy', 'torch', 'jax', 'cupy', 'tensorflow', 'pyclesperanto']
+          for fw in frameworks:
+              try:
+                  __import__(fw)
+                  print(f'✓ {fw} available')
+              except ImportError:
+                  print(f'✗ {fw} not available (will be skipped)')
+          "
+
+      - name: Run comprehensive GPU cleanup tests
+        run: |
+          # Run all GPU cleanup tests
+          # Tests will use frameworks if available, skip gracefully if not
+          pytest -v tests/test_gpu_cleanup.py \
+            --cov=arraybridge \
+            --cov-report=term-missing \
+            --cov-report=html \
+            --tb=short \
+            -ra
+
+      - name: Run framework-specific GPU tests
+        run: |
+          # Run tests marked for specific frameworks
+          pytest -v tests/ -k "gpu or cupy or torch or tensorflow or jax or pyclesperanto" \
+            --cov=arraybridge \
+            --cov-report=term-missing \
+            -ra || true
+
+      - name: Test results summary
+        if: always()
+        run: |
+          echo "GPU Testing Complete!"
+          echo "Note: Full GPU testing requires NVIDIA CUDA infrastructure."
+          echo "For complete GPU testing, use a system with NVIDIA GPUs installed."
+
+      - name: Upload coverage report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: gpu-test-coverage-report
+          path: |
+            htmlcov/
+            .coverage
+

arraybridge-0.2.9/ADDING_NEW_FRAMEWORKS.md

@@ -0,0 +1,215 @@
+# Adding New Frameworks to arraybridge
+
+With the metaclass-registry integration, adding a new framework is now simpler and requires no manual wiring.
+
+## Quick Overview
+
+To add a new framework (e.g., MXNet), you need to:
+
+1. Add the new memory type to the `MemoryType` enum in `types.py`
+2. Add framework configuration to `_FRAMEWORK_CONFIG` in `framework_config.py`
+3. The converter automatically registers itself - no manual registration needed!
+
+## Step-by-Step Guide
+
+### Step 1: Add to MemoryType Enum
+
+Edit `src/arraybridge/types.py`:
+
+```python
+class MemoryType(Enum):
+    """Enum representing different array/tensor framework types."""
+    
+    NUMPY = "numpy"
+    CUPY = "cupy"
+    TORCH = "torch"
+    TENSORFLOW = "tensorflow"
+    JAX = "jax"
+    PYCLESPERANTO = "pyclesperanto"
+    MXNET = "mxnet"  # <-- Add your new framework
+```
+
+### Step 2: Add Framework Configuration
+
+Edit `src/arraybridge/framework_config.py` and add a new entry to `_FRAMEWORK_CONFIG`:
+
+```python
+_FRAMEWORK_CONFIG = {
+    # ... existing configurations ...
+    
+    MemoryType.MXNET: {
+        # Metadata
+        'import_name': 'mxnet',
+        'display_name': 'MXNet',
+        'is_gpu': True,
+        
+        # Conversion operations - these define the converter methods
+        'conversion_ops': {
+            'to_numpy': 'data.asnumpy()',  # How to convert to numpy
+            'from_numpy': '{mod}.nd.array(data, ctx={mod}.gpu(gpu_id))',  # How to create from numpy
+            'from_dlpack': '{mod}.nd.from_dlpack(data)',  # DLPack support (if available)
+            'move_to_device': 'data.as_in_context({mod}.gpu(device_id))',  # Move between devices
+        },
+        
+        # Device operations (optional)
+        'get_device_id': 'data.context.device_id',
+        'set_device': None,
+        
+        # Other configuration...
+        'supports_dlpack': True,
+        'validate_dlpack': None,
+        
+        # ... add other required fields based on existing frameworks
+    }
+}
+```
+
+### Step 3: That's It!
+
+The converter class is automatically created and registered when the module loads. You can verify it works:
+
+```python
+from arraybridge.converters_registry import ConverterBase, get_converter
+from arraybridge.types import MemoryType
+
+# Check that it's registered
+print(sorted(ConverterBase.__registry__.keys()))
+# Output: ['cupy', 'jax', 'mxnet', 'numpy', 'pyclesperanto', 'tensorflow', 'torch']
+
+# Get the converter
+mxnet_converter = get_converter("mxnet")
+print(mxnet_converter.memory_type)  # Output: 'mxnet'
+
+# Use via MemoryType enum
+converter = MemoryType.MXNET.converter
+```
+
+## What Happens Behind the Scenes
+
+1. **Auto-generation**: A `MxnetConverter` class is created dynamically with methods from `conversion_ops`
+2. **Auto-registration**: The metaclass `AutoRegisterMeta` automatically registers it in `ConverterBase.__registry__`
+3. **Auto-validation**: Module load validates that all `MemoryType` values have registered converters
+4. **Auto-methods**: The converter automatically gets `to_X()` methods for all other frameworks
+
+## Benefits of This Approach
+
+### Before (Manual Wiring - Old System)
+```python
+# Had to manually create converter class with all methods
+class MxnetConverter(MemoryTypeConverter):
+    def to_numpy(self, data, gpu_id):
+        return data.asnumpy()
+    def from_numpy(self, data, gpu_id):
+        return mxnet.nd.array(data)
+    def to_torch(self, data, gpu_id):
+        # ... manual implementation
+    def to_cupy(self, data, gpu_id):
+        # ... manual implementation
+    # ... 6+ more methods
+
+# Had to manually register
+_CONVERTERS[MemoryType.MXNET] = MxnetConverter()
+```
+
+### After (Auto-registration - New System)
+```python
+# Just add to enum and config - everything else is automatic!
+MemoryType.MXNET = "mxnet"
+
+_FRAMEWORK_CONFIG[MemoryType.MXNET] = {
+    'conversion_ops': {
+        'to_numpy': 'data.asnumpy()',
+        # ...
+    }
+}
+```
+
+## Framework Configuration Reference
+
+Required fields in `conversion_ops`:
+- `to_numpy`: Expression to convert framework data to numpy
+- `from_numpy`: Expression to create framework data from numpy
+- `from_dlpack`: Expression for DLPack conversion (or `None`)
+- `move_to_device`: Expression to move data between devices
+
+Available template variables in expressions:
+- `{mod}`: The imported module (e.g., `mxnet`)
+- `data`: The input data
+- `gpu_id` / `device_id`: Target device ID
+
+## Testing Your New Framework
+
+Add tests in `tests/test_converters.py`:
+
+```python
+@pytest.mark.mxnet
+def test_convert_numpy_to_mxnet(self, mxnet_available):
+    """Test converting NumPy to MXNet."""
+    if not mxnet_available:
+        pytest.skip("MXNet not available")
+    
+    import mxnet as mx
+    arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)
+    result = convert_memory(arr, source_type="numpy", target_type="mxnet", gpu_id=0)
+    
+    assert isinstance(result, mx.nd.NDArray)
+    np.testing.assert_array_almost_equal(result.asnumpy(), arr)
+```
+
+## Common Patterns
+
+### GPU Framework with DLPack
+```python
+'conversion_ops': {
+    'to_numpy': 'data.cpu().numpy()',
+    'from_numpy': '{mod}.from_numpy(data).to(device=gpu_id)',
+    'from_dlpack': '{mod}.from_dlpack(data)',
+    'move_to_device': 'data.to(device=device_id)',
+}
+```
+
+### CPU-only Framework
+```python
+'conversion_ops': {
+    'to_numpy': 'np.array(data)',
+    'from_numpy': '{mod}.array(data)',
+    'from_dlpack': None,  # Not supported
+    'move_to_device': 'data',  # No-op for CPU
+}
+```
+
+### Complex Operations with Helpers
+If you need complex logic, define a helper function in `framework_config.py`:
+
+```python
+def _mxnet_special_conversion(data, gpu_id, mod):
+    # Complex logic here
+    return result
+
+_FRAMEWORK_CONFIG[MemoryType.MXNET] = {
+    'conversion_ops': {
+        'from_numpy': _mxnet_special_conversion,  # Use callable instead of string
+        # ...
+    }
+}
+```
+
+## Troubleshooting
+
+### Converter not registered
+Make sure:
+1. You added the framework to `MemoryType` enum
+2. You added configuration to `_FRAMEWORK_CONFIG`
+3. The key in `_FRAMEWORK_CONFIG` matches the `MemoryType` enum value
+
+### Import errors
+If you get import errors, check:
+1. The `import_name` matches the actual package name
+2. The `conversion_ops` expressions use correct module syntax
+
+### Validation errors
+Run this to check registration:
+```python
+from arraybridge.converters_registry import _validate_registry
+_validate_registry()  # Raises RuntimeError if validation fails
+```

arraybridge-0.2.9/CI_ARTIFACT_UPDATE.md

@@ -0,0 +1,47 @@
+# GitHub Actions upload-artifact v3 → v4 Migration
+
+## Issue
+GitHub deprecated `actions/upload-artifact@v3` effective April 16, 2024. The CI workflows were failing with:
+```
+Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v3`.
+```
+
+## Solution
+Updated all `upload-artifact` action references from `v3` to `v4` in CI workflows.
+
+## Files Updated
+
+### 1. `.github/workflows/ci.yml`
+**Line 91**: Updated GPU test artifact upload
+```yaml
+# Before
+uses: actions/upload-artifact@v3
+
+# After
+uses: actions/upload-artifact@v4
+```
+
+### 2. `.github/workflows/gpu-tests.yml`
+**Line 41**: Updated standalone GPU test artifact upload
+```yaml
+# Before
+uses: actions/upload-artifact@v3
+
+# After
+uses: actions/upload-artifact@v4
+```
+
+## Changes Made
+- ✅ Both workflow files now use `actions/upload-artifact@v4`
+- ✅ Artifact upload configuration remains the same
+- ✅ CI workflows will no longer fail due to deprecated action
+
+## Reference
+- [GitHub Blog: Deprecation Notice for Artifact Actions](https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/)
+- [Upload Artifact v4 Documentation](https://github.com/actions/upload-artifact/releases/tag/v4)
+
+## Verification
+The GPU test CI job should now:
+1. ✅ Run without the deprecation error
+2. ✅ Successfully upload test results and coverage reports
+3. ✅ Display artifacts in the GitHub Actions UI