atlan-application-sdk 0.1.1rc38__py3-none-any.whl → 0.1.1rc40__py3-none-any.whl

application_sdk/decorators/.cursor/BUGBOT.md

@@ -0,0 +1,279 @@
+# Decorator Code Review Guidelines - Centralized Function Decorators
+
+## Context-Specific Patterns
+
+This directory contains all decorator implementations for the Application SDK. Decorators must be centralized here to avoid scattered functionality and ensure consistent patterns.
+
+### Phase 1: Critical Decorator Safety Issues
+
+**Decorator Centralization:**
+
+- **ALL decorators must be in this directory**: No decorators should exist in other modules (lock/, observability/, etc.)
+- **Consolidate scattered decorators**: If decorators are found elsewhere, they must be moved here
+- **Single responsibility per file**: Each decorator type should have its own file (locks.py, observability_decorator.py)
+- **Proper imports**: Other modules should import decorators from here, not define their own
+
+**Type Safety and Function Signatures:**
+
+- All decorators must preserve function signatures and type hints
+- Use `functools.wraps` to maintain function metadata
+- Generic decorators must use proper type annotations
+- Return types must match the original function's return type
+
+```python
+# ✅ DO: Proper decorator type safety
+from typing import Callable, Any, TypeVar, ParamSpec
+from functools import wraps
+
+P = ParamSpec('P')
+T = TypeVar('T')
+
+def my_decorator(func: Callable[P, T]) -> Callable[P, T]:
+    @wraps(func)
+    def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
+        # Decorator logic
+        return func(*args, **kwargs)
+    return wrapper
+
+# ❌ NEVER: Poor type annotations
+def bad_decorator(func):  # No type hints
+    def wrapper(*args, **kwargs):  # No type preservation
+        return func(*args, **kwargs)
+    return wrapper  # Missing @wraps
+```
+
+### Phase 2: Decorator Architecture Patterns
+
+**Proper Decorator Structure:**
+
+- **Parameterized decorators**: Support both `@decorator` and `@decorator(param=value)` usage patterns
+- **Error handling**: Decorators must not swallow exceptions unless explicitly designed to do so
+- **Resource cleanup**: Decorators that acquire resources must ensure cleanup in finally blocks
+- **Context preservation**: Maintain original function context and metadata
+
+**Configuration Management:**
+
+- **Centralized constants**: All decorator configuration should use constants from this directory
+- **Shared configuration**: Related decorators should share configuration patterns
+- **Environment awareness**: Decorators should work in both development and production environments
+
+```python
+# ✅ DO: Proper decorator configuration
+from application_sdk.constants import DEFAULT_LOCK_TTL, DEFAULT_MAX_LOCKS
+
+# Shared configuration for lock decorators
+LOCK_CONFIG_KEY = "distributed_lock_config"  # Centralized key
+
+def distributed_lock(
+    lock_name: Optional[str] = None,
+    max_locks: int = DEFAULT_MAX_LOCKS,
+    ttl_seconds: int = DEFAULT_LOCK_TTL
+):
+    """Distributed lock decorator with proper defaults and configuration."""
+
+    def decorator(func: Callable[P, T]) -> Callable[P, T]:
+        @wraps(func)
+        async def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
+            # Use centralized configuration
+            actual_lock_name = lock_name or f"{func.__module__}.{func.__name__}"
+
+            # Store config in activity context using shared key
+            activity_info = activity.info()
+            activity_info.memo[LOCK_CONFIG_KEY] = {
+                "lock_name": actual_lock_name,
+                "max_locks": max_locks,
+                "ttl_seconds": ttl_seconds
+            }
+
+            return await func(*args, **kwargs)
+        return wrapper
+    return decorator
+
+# ❌ REJECT: Scattered constants and configuration
+def bad_lock_decorator(max_locks=10):  # Hardcoded default
+    LOCK_KEY = "my_lock_key"  # Should be centralized
+    # Configuration scattered across files
+```
+
+### Phase 3: Decorator Testing Requirements
+
+**Comprehensive Decorator Testing:**
+
+- **Function preservation**: Test that decorators preserve original function behavior
+- **Type safety**: Verify type hints are maintained after decoration
+- **Error propagation**: Ensure exceptions are properly handled and propagated
+- **Resource cleanup**: Test cleanup behavior in both success and failure cases
+- **Configuration validation**: Test all configuration parameters and edge cases
+
+```python
+# ✅ DO: Comprehensive decorator testing
+@pytest.mark.asyncio
+class TestDistributedLockDecorator:
+    """Test suite for distributed lock decorator."""
+
+    async def test_function_signature_preservation(self):
+        """Test that decorator preserves function signature and types."""
+
+        @distributed_lock("test_lock")
+        async def test_function(param1: str, param2: int = 10) -> dict:
+            """Test function docstring."""
+            return {"param1": param1, "param2": param2}
+
+        # Verify signature preservation
+        assert test_function.__name__ == "test_function"
+        assert test_function.__doc__ == "Test function docstring."
+
+        # Verify function still works
+        result = await test_function("test", 20)
+        assert result == {"param1": "test", "param2": 20}
+
+    async def test_error_propagation(self):
+        """Test that decorator properly propagates exceptions."""
+
+        @distributed_lock("error_lock")
+        async def failing_function():
+            raise ValueError("Test error")
+
+        # Verify exception is propagated, not swallowed
+        with pytest.raises(ValueError, match="Test error"):
+            await failing_function()
+
+    async def test_resource_cleanup_on_failure(self, mock_lock_manager):
+        """Test that resources are cleaned up even when function fails."""
+
+        @distributed_lock("cleanup_test")
+        async def failing_function():
+            raise RuntimeError("Simulated failure")
+
+        mock_lock_manager.acquire_lock.return_value.__aenter__ = AsyncMock()
+        mock_lock_manager.acquire_lock.return_value.__aexit__ = AsyncMock()
+
+        with pytest.raises(RuntimeError):
+            await failing_function()
+
+        # Verify cleanup was called
+        mock_lock_manager.acquire_lock.return_value.__aexit__.assert_called_once()
+```
+
+### Phase 4: Performance and Integration
+
+**Decorator Performance:**
+
+- **Minimal overhead**: Decorators should add minimal performance overhead
+- **Async compatibility**: All decorators must work correctly with async functions
+- **Context manager efficiency**: Use efficient context managers for resource management
+- **Caching**: Cache expensive decorator setup operations where appropriate
+
+**Integration Patterns:**
+
+- **Temporal integration**: Decorators must work correctly with Temporal activities and workflows
+- **Observability integration**: Integrate with logging, metrics, and tracing systems
+- **Error handling integration**: Work correctly with the SDK's error handling patterns
+
+### Phase 5: Decorator Maintainability
+
+**Code Organization:**
+
+- **One decorator type per file**: Keep related decorators together (all lock decorators in locks.py)
+- **Clear naming**: Decorator files should clearly indicate their purpose
+- **Consistent patterns**: All decorators should follow the same structural patterns
+- **Documentation**: Each decorator must have comprehensive docstrings with usage examples
+
+**Backwards Compatibility:**
+
+- **API stability**: Decorator APIs should be stable across versions
+- **Graceful deprecation**: Deprecated decorators should include migration guidance
+- **Version compatibility**: Support existing usage patterns when adding new features
+
+---
+
+## Decorator-Specific Anti-Patterns
+
+**Always Reject:**
+
+- **Scattered decorators**: Decorators defined outside this directory
+- **Missing type safety**: Decorators without proper type annotations
+- **Resource leaks**: Decorators that don't clean up resources properly
+- **Exception swallowing**: Decorators that hide exceptions unintentionally
+- **Poor configuration**: Hardcoded values that should be configurable
+- **No function preservation**: Decorators that don't preserve original function metadata
+
+**Centralization Anti-Patterns:**
+
+```python
+# ❌ REJECT: Decorators in wrong locations
+# Found in application_sdk/lock/__init__.py
+def needs_lock(max_locks=10):
+    """Should be in decorators/locks.py instead"""
+
+# Found in application_sdk/observability/some_module.py
+def trace_activity(func):
+    """Should be in decorators/observability_decorator.py"""
+
+# ✅ REQUIRE: Centralized decorators
+# In application_sdk/decorators/locks.py
+def needs_lock(max_locks: int = DEFAULT_MAX_LOCKS):
+    """Properly located distributed lock decorator"""
+
+# In application_sdk/decorators/observability_decorator.py
+def observability(logger=None, metrics=None, traces=None):
+    """Properly located observability decorator"""
+```
+
+**Type Safety Anti-Patterns:**
+
+```python
+# ❌ REJECT: Poor type safety
+def bad_decorator(func):  # No type annotations
+    def wrapper(*args, **kwargs):  # No parameter specifications
+        return func(*args, **kwargs)
+    return wrapper  # Missing @wraps, no return type
+
+# ✅ REQUIRE: Proper type safety
+from typing import Callable, TypeVar, ParamSpec
+from functools import wraps
+
+P = ParamSpec('P')
+T = TypeVar('T')
+
+def good_decorator(func: Callable[P, T]) -> Callable[P, T]:
+    @wraps(func)
+    def wrapper(*args: P.args, **kwargs: P.kwargs) -> T:
+        return func(*args, **kwargs)
+    return wrapper
+```
+
+**Configuration Anti-Patterns:**
+
+```python
+# ❌ REJECT: Scattered configuration
+# Different files using different keys for same concept
+LOCK_KEY_1 = "lock_config"  # In locks.py
+LOCK_KEY_2 = "distributed_lock"  # In interceptors.py
+DEFAULT_TTL = 300  # Hardcoded in decorator
+
+# ✅ REQUIRE: Centralized configuration
+# In application_sdk/constants.py
+DISTRIBUTED_LOCK_CONFIG_KEY = "distributed_lock_config"
+DEFAULT_LOCK_TTL = 300
+DEFAULT_MAX_LOCKS = 10
+
+# In decorators using shared constants
+from application_sdk.constants import DISTRIBUTED_LOCK_CONFIG_KEY, DEFAULT_LOCK_TTL
+```
+
+## Educational Context for Decorator Reviews
+
+When reviewing decorator code, emphasize:
+
+1. **Centralization Impact**: "Scattered decorators create maintenance nightmares. When the same decorator logic appears in multiple places, bugs get fixed in some places but not others. Centralization ensures consistency and reduces maintenance burden."
+
+2. **Type Safety Impact**: "Decorators that don't preserve type information break IDE support, static analysis, and developer productivity. Proper type annotations are essential for maintaining code quality in large codebases."
+
+3. **Resource Management Impact**: "Decorators often manage resources (locks, connections, contexts). Poor resource management in decorators can cause system-wide issues because they're used across many functions."
+
+4. **Function Preservation Impact**: "Decorators that don't preserve original function metadata break debugging, introspection, and documentation tools. Using @functools.wraps is not optional."
+
+5. **Testing Impact**: "Decorators are cross-cutting concerns that affect many functions. Bugs in decorators have amplified impact, making thorough testing especially critical."
+
+6. **Performance Impact**: "Decorators add overhead to every function call they wrap. Inefficient decorators can degrade system performance across the entire application."

application_sdk/inputs/.cursor/BUGBOT.md

@@ -0,0 +1,250 @@
+# Input Code Review Guidelines - Data Input Processing
+
+## Context-Specific Patterns
+
+This directory contains input processing implementations for various data formats (JSON, Parquet, SQL). Input processors must handle data efficiently while maintaining data integrity and performance.
+
+### Phase 1: Critical Input Safety Issues
+
+**Object Store Path Management:**
+
+- **Correct path calculation**: Source paths must use the actual object store prefix, not derived local paths
+- **Path validation**: Verify that object store keys are valid and within constraints
+- **User-provided prefixes**: Respect user-configured input prefixes and download paths
+- **Path consistency**: Ensure downloaded files match the expected object store locations
+
+**Data Validation and Security:**
+
+- All input data must be validated before processing
+- File size limits must be enforced to prevent resource exhaustion
+- File type validation required for uploaded/downloaded files
+- Malicious file content detection for executable or script files
+- Input path traversal prevention
+
+```python
+# ✅ DO: Proper object store path handling
+class JsonInput:
+    async def download_from_object_store(
+        self,
+        input_prefix: str,  # User-provided prefix
+        local_destination: str
+    ) -> List[str]:
+        """Download files with correct path handling."""
+
+        # Use the actual input prefix, not derived local path
+        object_store_source = input_prefix  # Keep user's intended source
+
+        downloaded_files = await self.object_store.download_files(
+            source=object_store_source,
+            destination=local_destination
+        )
+
+        return downloaded_files
+
+# ❌ REJECT: Incorrect path handling
+class BadJsonInput:
+    async def download_from_object_store(
+        self,
+        input_prefix: str,
+        local_destination: str
+    ) -> List[str]:
+        # Wrong: derives object store path from local path
+        object_store_source = get_object_store_prefix(local_destination)
+        # This ignores the user's actual input_prefix!
+
+        return await self.object_store.download_files(
+            source=object_store_source,  # Wrong source!
+            destination=local_destination
+        )
+```
+
+### Phase 2: Input Architecture Patterns
+
+**Performance Optimization Requirements:**
+
+- **Parallelization opportunities**: Flag sequential file operations that could be parallelized
+- **Batch processing**: Group related operations to reduce overhead
+- **Memory efficiency**: Process large files in chunks, not all at once
+- **Connection reuse**: Optimize object store connections across operations
+
+**Resource Management:**
+
+- Use proper connection pooling for object store operations
+- Implement timeout handling for download operations
+- Clean up temporary files after processing
+- Handle partial download failures gracefully
+- Monitor memory usage during large file processing
+
+```python
+# ✅ DO: Parallelized file processing
+async def download_multiple_files_parallel(
+    self,
+    file_paths: List[str],
+    destination_dir: str
+) -> List[str]:
+    """Download multiple files in parallel for better performance."""
+
+    async def download_single_file(file_path: str) -> str:
+        """Download a single file with error handling."""
+        try:
+            return await self.object_store.download_file(
+                source=file_path,
+                destination=os.path.join(destination_dir, os.path.basename(file_path))
+            )
+        except Exception as e:
+            logger.error(f"Failed to download {file_path}: {e}")
+            raise
+
+    # Parallel processing with controlled concurrency
+    semaphore = asyncio.Semaphore(10)  # Limit concurrent downloads
+
+    async def download_with_semaphore(file_path: str) -> str:
+        async with semaphore:
+            return await download_single_file(file_path)
+
+    tasks = [download_with_semaphore(path) for path in file_paths]
+    return await asyncio.gather(*tasks)
+
+# ❌ REJECT: Sequential processing
+async def download_multiple_files_sequential(self, file_paths: List[str]) -> List[str]:
+    """Sequential download - should be flagged for parallelization."""
+    downloaded = []
+    for file_path in file_paths:  # FLAG: Could be parallelized
+        result = await self.object_store.download_file(file_path)
+        downloaded.append(result)
+    return downloaded
+```
+
+### Phase 3: Input Testing Requirements
+
+**Data Input Testing:**
+
+- Test with various file formats and sizes
+- Test malformed data handling
+- Test partial download/upload scenarios
+- Mock object store operations in unit tests
+- Include integration tests with real object store
+- Test error recovery and retry logic
+
+**Performance Testing:**
+
+- Include tests for large file processing
+- Test memory usage with different chunk sizes
+- Test concurrent download/upload operations
+- Verify timeout handling works correctly
+- Test connection pool behavior
+
+### Phase 4: Performance and Scalability
+
+**Data Processing Efficiency:**
+
+- Use streaming for large files instead of loading entirely into memory
+- Implement proper chunking for batch operations
+- Use async generators for memory-efficient data processing
+- Monitor memory usage and processing time
+- Optimize file I/O operations
+
+**Object Store Optimization:**
+
+- Use connection pooling for object store clients
+- Implement proper retry logic for transient failures
+- Use parallel operations where appropriate
+- Cache frequently accessed metadata
+- Monitor object store operation metrics
+
+### Phase 5: Input Data Maintainability
+
+**Error Handling and Recovery:**
+
+- Implement comprehensive error handling for all input operations
+- Provide meaningful error messages with context
+- Handle partial failures gracefully (some files fail, others succeed)
+- Implement proper retry logic for transient failures
+- Log all input operations with sufficient context
+
+**Configuration Management:**
+
+- Externalize all input-related configuration
+- Support different input sources and formats
+- Validate input configuration before processing
+- Document all supported input parameters
+- Handle environment-specific input requirements
+
+---
+
+## Input-Specific Anti-Patterns
+
+**Always Reject:**
+
+- **Path calculation errors**: Using local paths to derive object store paths
+- **Sequential processing**: Processing multiple files sequentially when parallel processing is possible
+- **Memory inefficiency**: Loading large files entirely into memory
+- **Missing error handling**: Input operations without proper try-catch blocks
+- **Poor path validation**: Not validating object store keys or file paths
+- **Resource leaks**: Not cleaning up temporary files or connections
+
+**Object Store Anti-Patterns:**
+
+```python
+# ❌ REJECT: Incorrect object store usage
+class BadInputProcessor:
+    async def process_files(self, local_files: List[str]):
+        # Wrong: derives object store path from local path
+        for local_file in local_files:
+            object_store_key = get_object_store_prefix(local_file)  # Incorrect!
+            await self.object_store.download_file(object_store_key, local_file)
+
+# ✅ REQUIRE: Correct object store usage
+class GoodInputProcessor:
+    async def process_files(
+        self,
+        object_store_paths: List[str],  # Actual object store paths
+        local_destination_dir: str
+    ):
+        # Use actual object store paths, not derived ones
+        for object_store_path in object_store_paths:
+            local_file = os.path.join(
+                local_destination_dir,
+                os.path.basename(object_store_path)
+            )
+            await self.object_store.download_file(object_store_path, local_file)
+```
+
+**Performance Anti-Patterns:**
+
+```python
+# ❌ REJECT: Sequential file processing
+async def process_files_sequential(file_list: List[str]):
+    results = []
+    for file_path in file_list:  # Should be parallelized
+        result = await process_single_file(file_path)
+        results.append(result)
+    return results
+
+# ✅ REQUIRE: Parallel file processing
+async def process_files_parallel(file_list: List[str], max_concurrency: int = 10):
+    semaphore = asyncio.Semaphore(max_concurrency)
+
+    async def process_with_semaphore(file_path: str):
+        async with semaphore:
+            return await process_single_file(file_path)
+
+    tasks = [process_with_semaphore(path) for path in file_list]
+    return await asyncio.gather(*tasks, return_exceptions=True)
+```
+
+## Educational Context for Input Reviews
+
+When reviewing input code, emphasize:
+
+1. **Data Integrity Impact**: "Incorrect object store path handling can cause data loss or corruption. Files uploaded to wrong locations become inaccessible, breaking data processing pipelines."
+
+2. **Performance Impact**: "Sequential file processing creates unnecessary bottlenecks. For enterprise datasets with hundreds of files, parallelization can reduce processing time from hours to minutes."
+
+3. **Resource Impact**: "Poor memory management in input processing can cause out-of-memory errors with large datasets. Streaming and chunking are essential for enterprise-scale data processing."
+
+4. **User Experience Impact**: "Input path handling errors are often silent until runtime, causing difficult-to-debug failures. Proper validation and clear error messages save hours of troubleshooting."
+
+5. **Scalability Impact**: "Input processing patterns that work for small datasets can fail catastrophically at enterprise scale. Always design for the largest expected dataset size."
+
+6. **Reliability Impact**: "Input operations are often the first point of failure in data pipelines. Robust error handling and retry logic in input processing prevents entire workflows from failing due to transient issues."