@bgicli/bgicli 2.1.1 → 2.2.0

package/data/skills/zarr-python/SKILL.md

@@ -0,0 +1,777 @@
+---
+name: zarr-python
+description: Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.
+license: MIT license
+metadata:
+    skill-author: K-Dense Inc.
+---
+
+# Zarr Python
+
+## Overview
+
+Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.
+
+## Quick Start
+
+### Installation
+
+```bash
+uv pip install zarr
+```
+
+Requires Python 3.11+. For cloud storage support, install additional packages:
+```python
+uv pip install s3fs  # For S3
+uv pip install gcsfs  # For Google Cloud Storage
+```
+
+### Basic Array Creation
+
+```python
+import zarr
+import numpy as np
+
+# Create a 2D array with chunking and compression
+z = zarr.create_array(
+    store="data/my_array.zarr",
+    shape=(10000, 10000),
+    chunks=(1000, 1000),
+    dtype="f4"
+)
+
+# Write data using NumPy-style indexing
+z[:, :] = np.random.random((10000, 10000))
+
+# Read data
+data = z[0:100, 0:100]  # Returns NumPy array
+```
+
+## Core Operations
+
+### Creating Arrays
+
+Zarr provides multiple convenience functions for array creation:
+
+```python
+# Create empty array
+z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',
+               store='data.zarr')
+
+# Create filled arrays
+z = zarr.ones((5000, 5000), chunks=(500, 500))
+z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))
+
+# Create from existing data
+data = np.arange(10000).reshape(100, 100)
+z = zarr.array(data, chunks=(10, 10), store='data.zarr')
+
+# Create like another array
+z2 = zarr.zeros_like(z)  # Matches shape, chunks, dtype of z
+```
+
+### Opening Existing Arrays
+
+```python
+# Open array (read/write mode by default)
+z = zarr.open_array('data.zarr', mode='r+')
+
+# Read-only mode
+z = zarr.open_array('data.zarr', mode='r')
+
+# The open() function auto-detects arrays vs groups
+z = zarr.open('data.zarr')  # Returns Array or Group
+```
+
+### Reading and Writing Data
+
+Zarr arrays support NumPy-like indexing:
+
+```python
+# Write entire array
+z[:] = 42
+
+# Write slices
+z[0, :] = np.arange(100)
+z[10:20, 50:60] = np.random.random((10, 10))
+
+# Read data (returns NumPy array)
+data = z[0:100, 0:100]
+row = z[5, :]
+
+# Advanced indexing
+z.vindex[[0, 5, 10], [2, 8, 15]]  # Coordinate indexing
+z.oindex[0:10, [5, 10, 15]]       # Orthogonal indexing
+z.blocks[0, 0]                     # Block/chunk indexing
+```
+
+### Resizing and Appending
+
+```python
+# Resize array
+z.resize(15000, 15000)  # Expands or shrinks dimensions
+
+# Append data along an axis
+z.append(np.random.random((1000, 10000)), axis=0)  # Adds rows
+```
+
+## Chunking Strategies
+
+Chunking is critical for performance. Choose chunk sizes and shapes based on access patterns.
+
+### Chunk Size Guidelines
+
+- **Minimum chunk size**: 1 MB recommended for optimal performance
+- **Balance**: Larger chunks = fewer metadata operations; smaller chunks = better parallel access
+- **Memory consideration**: Entire chunks must fit in memory during compression
+
+```python
+# Configure chunk size (aim for ~1MB per chunk)
+# For float32 data: 1MB = 262,144 elements = 512×512 array
+z = zarr.zeros(
+    shape=(10000, 10000),
+    chunks=(512, 512),  # ~1MB chunks
+    dtype='f4'
+)
+```
+
+### Aligning Chunks with Access Patterns
+
+**Critical**: Chunk shape dramatically affects performance based on how data is accessed.
+
+```python
+# If accessing rows frequently (first dimension)
+z = zarr.zeros((10000, 10000), chunks=(10, 10000))  # Chunk spans columns
+
+# If accessing columns frequently (second dimension)
+z = zarr.zeros((10000, 10000), chunks=(10000, 10))  # Chunk spans rows
+
+# For mixed access patterns (balanced approach)
+z = zarr.zeros((10000, 10000), chunks=(1000, 1000))  # Square chunks
+```
+
+**Performance example**: For a (200, 200, 200) array, reading along the first dimension:
+- Using chunks (1, 200, 200): ~107ms
+- Using chunks (200, 200, 1): ~1.65ms (65× faster!)
+
+### Sharding for Large-Scale Storage
+
+When arrays have millions of small chunks, use sharding to group chunks into larger storage objects:
+
+```python
+from zarr.codecs import ShardingCodec, BytesCodec
+from zarr.codecs.blosc import BloscCodec
+
+# Create array with sharding
+z = zarr.create_array(
+    store='data.zarr',
+    shape=(100000, 100000),
+    chunks=(100, 100),  # Small chunks for access
+    shards=(1000, 1000),  # Groups 100 chunks per shard
+    dtype='f4'
+)
+```
+
+**Benefits**:
+- Reduces file system overhead from millions of small files
+- Improves cloud storage performance (fewer object requests)
+- Prevents filesystem block size waste
+
+**Important**: Entire shards must fit in memory before writing.
+
+## Compression
+
+Zarr applies compression per chunk to reduce storage while maintaining fast access.
+
+### Configuring Compression
+
+```python
+from zarr.codecs.blosc import BloscCodec
+from zarr.codecs import GzipCodec, ZstdCodec
+
+# Default: Blosc with Zstandard
+z = zarr.zeros((1000, 1000), chunks=(100, 100))  # Uses default compression
+
+# Configure Blosc codec
+z = zarr.create_array(
+    store='data.zarr',
+    shape=(1000, 1000),
+    chunks=(100, 100),
+    dtype='f4',
+    codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]
+)
+
+# Available Blosc compressors: 'blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd'
+
+# Use Gzip compression
+z = zarr.create_array(
+    store='data.zarr',
+    shape=(1000, 1000),
+    chunks=(100, 100),
+    dtype='f4',
+    codecs=[GzipCodec(level=6)]
+)
+
+# Disable compression
+z = zarr.create_array(
+    store='data.zarr',
+    shape=(1000, 1000),
+    chunks=(100, 100),
+    dtype='f4',
+    codecs=[BytesCodec()]  # No compression
+)
+```
+
+### Compression Performance Tips
+
+- **Blosc** (default): Fast compression/decompression, good for interactive workloads
+- **Zstandard**: Better compression ratios, slightly slower than LZ4
+- **Gzip**: Maximum compression, slower performance
+- **LZ4**: Fastest compression, lower ratios
+- **Shuffle**: Enable shuffle filter for better compression on numeric data
+
+```python
+# Optimal for numeric scientific data
+codecs=[BloscCodec(cname='zstd', clevel=5, shuffle='shuffle')]
+
+# Optimal for speed
+codecs=[BloscCodec(cname='lz4', clevel=1)]
+
+# Optimal for compression ratio
+codecs=[GzipCodec(level=9)]
+```
+
+## Storage Backends
+
+Zarr supports multiple storage backends through a flexible storage interface.
+
+### Local Filesystem (Default)
+
+```python
+from zarr.storage import LocalStore
+
+# Explicit store creation
+store = LocalStore('data/my_array.zarr')
+z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
+
+# Or use string path (creates LocalStore automatically)
+z = zarr.open_array('data/my_array.zarr', mode='w', shape=(1000, 1000),
+                    chunks=(100, 100))
+```
+
+### In-Memory Storage
+
+```python
+from zarr.storage import MemoryStore
+
+# Create in-memory store
+store = MemoryStore()
+z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
+
+# Data exists only in memory, not persisted
+```
+
+### ZIP File Storage
+
+```python
+from zarr.storage import ZipStore
+
+# Write to ZIP file
+store = ZipStore('data.zip', mode='w')
+z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
+z[:] = np.random.random((1000, 1000))
+store.close()  # IMPORTANT: Must close ZipStore
+
+# Read from ZIP file
+store = ZipStore('data.zip', mode='r')
+z = zarr.open_array(store=store)
+data = z[:]
+store.close()
+```
+
+### Cloud Storage (S3, GCS)
+
+```python
+import s3fs
+import zarr
+
+# S3 storage
+s3 = s3fs.S3FileSystem(anon=False)  # Use credentials
+store = s3fs.S3Map(root='my-bucket/path/to/array.zarr', s3=s3)
+z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
+z[:] = data
+
+# Google Cloud Storage
+import gcsfs
+gcs = gcsfs.GCSFileSystem(project='my-project')
+store = gcsfs.GCSMap(root='my-bucket/path/to/array.zarr', gcs=gcs)
+z = zarr.open_array(store=store, mode='w', shape=(1000, 1000), chunks=(100, 100))
+```
+
+**Cloud Storage Best Practices**:
+- Use consolidated metadata to reduce latency: `zarr.consolidate_metadata(store)`
+- Align chunk sizes with cloud object sizing (typically 5-100 MB optimal)
+- Enable parallel writes using Dask for large-scale data
+- Consider sharding to reduce number of objects
+
+## Groups and Hierarchies
+
+Groups organize multiple arrays hierarchically, similar to directories or HDF5 groups.
+
+### Creating and Using Groups
+
+```python
+# Create root group
+root = zarr.group(store='data/hierarchy.zarr')
+
+# Create sub-groups
+temperature = root.create_group('temperature')
+precipitation = root.create_group('precipitation')
+
+# Create arrays within groups
+temp_array = temperature.create_array(
+    name='t2m',
+    shape=(365, 720, 1440),
+    chunks=(1, 720, 1440),
+    dtype='f4'
+)
+
+precip_array = precipitation.create_array(
+    name='prcp',
+    shape=(365, 720, 1440),
+    chunks=(1, 720, 1440),
+    dtype='f4'
+)
+
+# Access using paths
+array = root['temperature/t2m']
+
+# Visualize hierarchy
+print(root.tree())
+# Output:
+# /
+#  ├── temperature
+#  │   └── t2m (365, 720, 1440) f4
+#  └── precipitation
+#      └── prcp (365, 720, 1440) f4
+```
+
+### H5py-Compatible API
+
+Zarr provides an h5py-compatible interface for familiar HDF5 users:
+
+```python
+# Create group with h5py-style methods
+root = zarr.group('data.zarr')
+dataset = root.create_dataset('my_data', shape=(1000, 1000), chunks=(100, 100),
+                              dtype='f4')
+
+# Access like h5py
+grp = root.require_group('subgroup')
+arr = grp.require_dataset('array', shape=(500, 500), chunks=(50, 50), dtype='i4')
+```
+
+## Attributes and Metadata
+
+Attach custom metadata to arrays and groups using attributes:
+
+```python
+# Add attributes to array
+z = zarr.zeros((1000, 1000), chunks=(100, 100))
+z.attrs['description'] = 'Temperature data in Kelvin'
+z.attrs['units'] = 'K'
+z.attrs['created'] = '2024-01-15'
+z.attrs['processing_version'] = 2.1
+
+# Attributes are stored as JSON
+print(z.attrs['units'])  # Output: K
+
+# Add attributes to groups
+root = zarr.group('data.zarr')
+root.attrs['project'] = 'Climate Analysis'
+root.attrs['institution'] = 'Research Institute'
+
+# Attributes persist with the array/group
+z2 = zarr.open('data.zarr')
+print(z2.attrs['description'])
+```
+
+**Important**: Attributes must be JSON-serializable (strings, numbers, lists, dicts, booleans, null).
+
+## Integration with NumPy, Dask, and Xarray
+
+### NumPy Integration
+
+Zarr arrays implement the NumPy array interface:
+
+```python
+import numpy as np
+import zarr
+
+z = zarr.zeros((1000, 1000), chunks=(100, 100))
+
+# Use NumPy functions directly
+result = np.sum(z, axis=0)  # NumPy operates on Zarr array
+mean = np.mean(z[:100, :100])
+
+# Convert to NumPy array
+numpy_array = z[:]  # Loads entire array into memory
+```
+
+### Dask Integration
+
+Dask provides lazy, parallel computation on Zarr arrays:
+
+```python
+import dask.array as da
+import zarr
+
+# Create large Zarr array
+z = zarr.open('data.zarr', mode='w', shape=(100000, 100000),
+              chunks=(1000, 1000), dtype='f4')
+
+# Load as Dask array (lazy, no data loaded)
+dask_array = da.from_zarr('data.zarr')
+
+# Perform computations (parallel, out-of-core)
+result = dask_array.mean(axis=0).compute()  # Parallel computation
+
+# Write Dask array to Zarr
+large_array = da.random.random((100000, 100000), chunks=(1000, 1000))
+da.to_zarr(large_array, 'output.zarr')
+```
+
+**Benefits**:
+- Process datasets larger than memory
+- Automatic parallel computation across chunks
+- Efficient I/O with chunked storage
+
+### Xarray Integration
+
+Xarray provides labeled, multidimensional arrays with Zarr backend:
+
+```python
+import xarray as xr
+import zarr
+
+# Open Zarr store as Xarray Dataset (lazy loading)
+ds = xr.open_zarr('data.zarr')
+
+# Dataset includes coordinates and metadata
+print(ds)
+
+# Access variables
+temperature = ds['temperature']
+
+# Perform labeled operations
+subset = ds.sel(time='2024-01', lat=slice(30, 60))
+
+# Write Xarray Dataset to Zarr
+ds.to_zarr('output.zarr')
+
+# Create from scratch with coordinates
+ds = xr.Dataset(
+    {
+        'temperature': (['time', 'lat', 'lon'], data),
+        'precipitation': (['time', 'lat', 'lon'], data2)
+    },
+    coords={
+        'time': pd.date_range('2024-01-01', periods=365),
+        'lat': np.arange(-90, 91, 1),
+        'lon': np.arange(-180, 180, 1)
+    }
+)
+ds.to_zarr('climate_data.zarr')
+```
+
+**Benefits**:
+- Named dimensions and coordinates
+- Label-based indexing and selection
+- Integration with pandas for time series
+- NetCDF-like interface familiar to climate/geospatial scientists
+
+## Parallel Computing and Synchronization
+
+### Thread-Safe Operations
+
+```python
+from zarr import ThreadSynchronizer
+import zarr
+
+# For multi-threaded writes
+synchronizer = ThreadSynchronizer()
+z = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),
+                    chunks=(1000, 1000), synchronizer=synchronizer)
+
+# Safe for concurrent writes from multiple threads
+# (when writes don't span chunk boundaries)
+```
+
+### Process-Safe Operations
+
+```python
+from zarr import ProcessSynchronizer
+import zarr
+
+# For multi-process writes
+synchronizer = ProcessSynchronizer('sync_data.sync')
+z = zarr.open_array('data.zarr', mode='r+', shape=(10000, 10000),
+                    chunks=(1000, 1000), synchronizer=synchronizer)
+
+# Safe for concurrent writes from multiple processes
+```
+
+**Note**:
+- Concurrent reads require no synchronization
+- Synchronization only needed for writes that may span chunk boundaries
+- Each process/thread writing to separate chunks needs no synchronization
+
+## Consolidated Metadata
+
+For hierarchical stores with many arrays, consolidate metadata into a single file to reduce I/O operations:
+
+```python
+import zarr
+
+# After creating arrays/groups
+root = zarr.group('data.zarr')
+# ... create multiple arrays/groups ...
+
+# Consolidate metadata
+zarr.consolidate_metadata('data.zarr')
+
+# Open with consolidated metadata (faster, especially on cloud storage)
+root = zarr.open_consolidated('data.zarr')
+```
+
+**Benefits**:
+- Reduces metadata read operations from N (one per array) to 1
+- Critical for cloud storage (reduces latency)
+- Speeds up `tree()` operations and group traversal
+
+**Cautions**:
+- Metadata can become stale if arrays update without re-consolidation
+- Not suitable for frequently-updated datasets
+- Multi-writer scenarios may have inconsistent reads
+
+## Performance Optimization
+
+### Checklist for Optimal Performance
+
+1. **Chunk Size**: Aim for 1-10 MB per chunk
+   ```python
+   # For float32: 1MB = 262,144 elements
+   chunks = (512, 512)  # 512×512×4 bytes = ~1MB
+   ```
+
+2. **Chunk Shape**: Align with access patterns
+   ```python
+   # Row-wise access → chunk spans columns: (small, large)
+   # Column-wise access → chunk spans rows: (large, small)
+   # Random access → balanced: (medium, medium)
+   ```
+
+3. **Compression**: Choose based on workload
+   ```python
+   # Interactive/fast: BloscCodec(cname='lz4')
+   # Balanced: BloscCodec(cname='zstd', clevel=5)
+   # Maximum compression: GzipCodec(level=9)
+   ```
+
+4. **Storage Backend**: Match to environment
+   ```python
+   # Local: LocalStore (default)
+   # Cloud: S3Map/GCSMap with consolidated metadata
+   # Temporary: MemoryStore
+   ```
+
+5. **Sharding**: Use for large-scale datasets
+   ```python
+   # When you have millions of small chunks
+   shards=(10*chunk_size, 10*chunk_size)
+   ```
+
+6. **Parallel I/O**: Use Dask for large operations
+   ```python
+   import dask.array as da
+   dask_array = da.from_zarr('data.zarr')
+   result = dask_array.compute(scheduler='threads', num_workers=8)
+   ```
+
+### Profiling and Debugging
+
+```python
+# Print detailed array information
+print(z.info)
+
+# Output includes:
+# - Type, shape, chunks, dtype
+# - Compression codec and level
+# - Storage size (compressed vs uncompressed)
+# - Storage location
+
+# Check storage size
+print(f"Compressed size: {z.nbytes_stored / 1e6:.2f} MB")
+print(f"Uncompressed size: {z.nbytes / 1e6:.2f} MB")
+print(f"Compression ratio: {z.nbytes / z.nbytes_stored:.2f}x")
+```
+
+## Common Patterns and Best Practices
+
+### Pattern: Time Series Data
+
+```python
+# Store time series with time as first dimension
+# This allows efficient appending of new time steps
+z = zarr.open('timeseries.zarr', mode='a',
+              shape=(0, 720, 1440),  # Start with 0 time steps
+              chunks=(1, 720, 1440),  # One time step per chunk
+              dtype='f4')
+
+# Append new time steps
+new_data = np.random.random((1, 720, 1440))
+z.append(new_data, axis=0)
+```
+
+### Pattern: Large Matrix Operations
+
+```python
+import dask.array as da
+
+# Create large matrix in Zarr
+z = zarr.open('matrix.zarr', mode='w',
+              shape=(100000, 100000),
+              chunks=(1000, 1000),
+              dtype='f8')
+
+# Use Dask for parallel computation
+dask_z = da.from_zarr('matrix.zarr')
+result = (dask_z @ dask_z.T).compute()  # Parallel matrix multiply
+```
+
+### Pattern: Cloud-Native Workflow
+
+```python
+import s3fs
+import zarr
+
+# Write to S3
+s3 = s3fs.S3FileSystem()
+store = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)
+
+# Create array with appropriate chunking for cloud
+z = zarr.open_array(store=store, mode='w',
+                    shape=(10000, 10000),
+                    chunks=(500, 500),  # ~1MB chunks
+                    dtype='f4')
+z[:] = data
+
+# Consolidate metadata for faster reads
+zarr.consolidate_metadata(store)
+
+# Read from S3 (anywhere, anytime)
+store_read = s3fs.S3Map(root='s3://my-bucket/data.zarr', s3=s3)
+z_read = zarr.open_consolidated(store_read)
+subset = z_read[0:100, 0:100]
+```
+
+### Pattern: Format Conversion
+
+```python
+# HDF5 to Zarr
+import h5py
+import zarr
+
+with h5py.File('data.h5', 'r') as h5:
+    dataset = h5['dataset_name']
+    z = zarr.array(dataset[:],
+                   chunks=(1000, 1000),
+                   store='data.zarr')
+
+# NumPy to Zarr
+import numpy as np
+data = np.load('data.npy')
+z = zarr.array(data, chunks='auto', store='data.zarr')
+
+# Zarr to NetCDF (via Xarray)
+import xarray as xr
+ds = xr.open_zarr('data.zarr')
+ds.to_netcdf('data.nc')
+```
+
+## Common Issues and Solutions
+
+### Issue: Slow Performance
+
+**Diagnosis**: Check chunk size and alignment
+```python
+print(z.chunks)  # Are chunks appropriate size?
+print(z.info)    # Check compression ratio
+```
+
+**Solutions**:
+- Increase chunk size to 1-10 MB
+- Align chunks with access pattern
+- Try different compression codecs
+- Use Dask for parallel operations
+
+### Issue: High Memory Usage
+
+**Cause**: Loading entire array or large chunks into memory
+
+**Solutions**:
+```python
+# Don't load entire array
+# Bad: data = z[:]
+# Good: Process in chunks
+for i in range(0, z.shape[0], 1000):
+    chunk = z[i:i+1000, :]
+    process(chunk)
+
+# Or use Dask for automatic chunking
+import dask.array as da
+dask_z = da.from_zarr('data.zarr')
+result = dask_z.mean().compute()  # Processes in chunks
+```
+
+### Issue: Cloud Storage Latency
+
+**Solutions**:
+```python
+# 1. Consolidate metadata
+zarr.consolidate_metadata(store)
+z = zarr.open_consolidated(store)
+
+# 2. Use appropriate chunk sizes (5-100 MB for cloud)
+chunks = (2000, 2000)  # Larger chunks for cloud
+
+# 3. Enable sharding
+shards = (10000, 10000)  # Groups many chunks
+```
+
+### Issue: Concurrent Write Conflicts
+
+**Solution**: Use synchronizers or ensure non-overlapping writes
+```python
+from zarr import ProcessSynchronizer
+
+sync = ProcessSynchronizer('sync.sync')
+z = zarr.open_array('data.zarr', mode='r+', synchronizer=sync)
+
+# Or design workflow so each process writes to separate chunks
+```
+
+## Additional Resources
+
+For detailed API documentation, advanced usage, and the latest updates:
+
+- **Official Documentation**: https://zarr.readthedocs.io/
+- **Zarr Specifications**: https://zarr-specs.readthedocs.io/
+- **GitHub Repository**: https://github.com/zarr-developers/zarr-python
+- **Community Chat**: https://gitter.im/zarr-developers/community
+
+**Related Libraries**:
+- **Xarray**: https://docs.xarray.dev/ (labeled arrays)
+- **Dask**: https://docs.dask.org/ (parallel computing)
+- **NumCodecs**: https://numcodecs.readthedocs.io/ (compression codecs)
+