atlan-application-sdk 0.1.1rc39__py3-none-any.whl → 0.1.1rc41__py3-none-any.whl

application_sdk/inputs/.cursor/BUGBOT.md

@@ -0,0 +1,250 @@
+# Input Code Review Guidelines - Data Input Processing
+
+## Context-Specific Patterns
+
+This directory contains input processing implementations for various data formats (JSON, Parquet, SQL). Input processors must handle data efficiently while maintaining data integrity and performance.
+
+### Phase 1: Critical Input Safety Issues
+
+**Object Store Path Management:**
+
+- **Correct path calculation**: Source paths must use the actual object store prefix, not derived local paths
+- **Path validation**: Verify that object store keys are valid and within constraints
+- **User-provided prefixes**: Respect user-configured input prefixes and download paths
+- **Path consistency**: Ensure downloaded files match the expected object store locations
+
+**Data Validation and Security:**
+
+- All input data must be validated before processing
+- File size limits must be enforced to prevent resource exhaustion
+- File type validation required for uploaded/downloaded files
+- Malicious file content detection for executable or script files
+- Input path traversal prevention
+
+```python
+# ✅ DO: Proper object store path handling
+class JsonInput:
+    async def download_from_object_store(
+        self,
+        input_prefix: str,  # User-provided prefix
+        local_destination: str
+    ) -> List[str]:
+        """Download files with correct path handling."""
+
+        # Use the actual input prefix, not derived local path
+        object_store_source = input_prefix  # Keep user's intended source
+
+        downloaded_files = await self.object_store.download_files(
+            source=object_store_source,
+            destination=local_destination
+        )
+
+        return downloaded_files
+
+# ❌ REJECT: Incorrect path handling
+class BadJsonInput:
+    async def download_from_object_store(
+        self,
+        input_prefix: str,
+        local_destination: str
+    ) -> List[str]:
+        # Wrong: derives object store path from local path
+        object_store_source = get_object_store_prefix(local_destination)
+        # This ignores the user's actual input_prefix!
+
+        return await self.object_store.download_files(
+            source=object_store_source,  # Wrong source!
+            destination=local_destination
+        )
+```
+
+### Phase 2: Input Architecture Patterns
+
+**Performance Optimization Requirements:**
+
+- **Parallelization opportunities**: Flag sequential file operations that could be parallelized
+- **Batch processing**: Group related operations to reduce overhead
+- **Memory efficiency**: Process large files in chunks, not all at once
+- **Connection reuse**: Optimize object store connections across operations
+
+**Resource Management:**
+
+- Use proper connection pooling for object store operations
+- Implement timeout handling for download operations
+- Clean up temporary files after processing
+- Handle partial download failures gracefully
+- Monitor memory usage during large file processing
+
+```python
+# ✅ DO: Parallelized file processing
+async def download_multiple_files_parallel(
+    self,
+    file_paths: List[str],
+    destination_dir: str
+) -> List[str]:
+    """Download multiple files in parallel for better performance."""
+
+    async def download_single_file(file_path: str) -> str:
+        """Download a single file with error handling."""
+        try:
+            return await self.object_store.download_file(
+                source=file_path,
+                destination=os.path.join(destination_dir, os.path.basename(file_path))
+            )
+        except Exception as e:
+            logger.error(f"Failed to download {file_path}: {e}")
+            raise
+
+    # Parallel processing with controlled concurrency
+    semaphore = asyncio.Semaphore(10)  # Limit concurrent downloads
+
+    async def download_with_semaphore(file_path: str) -> str:
+        async with semaphore:
+            return await download_single_file(file_path)
+
+    tasks = [download_with_semaphore(path) for path in file_paths]
+    return await asyncio.gather(*tasks)
+
+# ❌ REJECT: Sequential processing
+async def download_multiple_files_sequential(self, file_paths: List[str]) -> List[str]:
+    """Sequential download - should be flagged for parallelization."""
+    downloaded = []
+    for file_path in file_paths:  # FLAG: Could be parallelized
+        result = await self.object_store.download_file(file_path)
+        downloaded.append(result)
+    return downloaded
+```
+
+### Phase 3: Input Testing Requirements
+
+**Data Input Testing:**
+
+- Test with various file formats and sizes
+- Test malformed data handling
+- Test partial download/upload scenarios
+- Mock object store operations in unit tests
+- Include integration tests with real object store
+- Test error recovery and retry logic
+
+**Performance Testing:**
+
+- Include tests for large file processing
+- Test memory usage with different chunk sizes
+- Test concurrent download/upload operations
+- Verify timeout handling works correctly
+- Test connection pool behavior
+
+### Phase 4: Performance and Scalability
+
+**Data Processing Efficiency:**
+
+- Use streaming for large files instead of loading entirely into memory
+- Implement proper chunking for batch operations
+- Use async generators for memory-efficient data processing
+- Monitor memory usage and processing time
+- Optimize file I/O operations
+
+**Object Store Optimization:**
+
+- Use connection pooling for object store clients
+- Implement proper retry logic for transient failures
+- Use parallel operations where appropriate
+- Cache frequently accessed metadata
+- Monitor object store operation metrics
+
+### Phase 5: Input Data Maintainability
+
+**Error Handling and Recovery:**
+
+- Implement comprehensive error handling for all input operations
+- Provide meaningful error messages with context
+- Handle partial failures gracefully (some files fail, others succeed)
+- Implement proper retry logic for transient failures
+- Log all input operations with sufficient context
+
+**Configuration Management:**
+
+- Externalize all input-related configuration
+- Support different input sources and formats
+- Validate input configuration before processing
+- Document all supported input parameters
+- Handle environment-specific input requirements
+
+---
+
+## Input-Specific Anti-Patterns
+
+**Always Reject:**
+
+- **Path calculation errors**: Using local paths to derive object store paths
+- **Sequential processing**: Processing multiple files sequentially when parallel processing is possible
+- **Memory inefficiency**: Loading large files entirely into memory
+- **Missing error handling**: Input operations without proper try-catch blocks
+- **Poor path validation**: Not validating object store keys or file paths
+- **Resource leaks**: Not cleaning up temporary files or connections
+
+**Object Store Anti-Patterns:**
+
+```python
+# ❌ REJECT: Incorrect object store usage
+class BadInputProcessor:
+    async def process_files(self, local_files: List[str]):
+        # Wrong: derives object store path from local path
+        for local_file in local_files:
+            object_store_key = get_object_store_prefix(local_file)  # Incorrect!
+            await self.object_store.download_file(object_store_key, local_file)
+
+# ✅ REQUIRE: Correct object store usage
+class GoodInputProcessor:
+    async def process_files(
+        self,
+        object_store_paths: List[str],  # Actual object store paths
+        local_destination_dir: str
+    ):
+        # Use actual object store paths, not derived ones
+        for object_store_path in object_store_paths:
+            local_file = os.path.join(
+                local_destination_dir,
+                os.path.basename(object_store_path)
+            )
+            await self.object_store.download_file(object_store_path, local_file)
+```
+
+**Performance Anti-Patterns:**
+
+```python
+# ❌ REJECT: Sequential file processing
+async def process_files_sequential(file_list: List[str]):
+    results = []
+    for file_path in file_list:  # Should be parallelized
+        result = await process_single_file(file_path)
+        results.append(result)
+    return results
+
+# ✅ REQUIRE: Parallel file processing
+async def process_files_parallel(file_list: List[str], max_concurrency: int = 10):
+    semaphore = asyncio.Semaphore(max_concurrency)
+
+    async def process_with_semaphore(file_path: str):
+        async with semaphore:
+            return await process_single_file(file_path)
+
+    tasks = [process_with_semaphore(path) for path in file_list]
+    return await asyncio.gather(*tasks, return_exceptions=True)
+```
+
+## Educational Context for Input Reviews
+
+When reviewing input code, emphasize:
+
+1. **Data Integrity Impact**: "Incorrect object store path handling can cause data loss or corruption. Files uploaded to wrong locations become inaccessible, breaking data processing pipelines."
+
+2. **Performance Impact**: "Sequential file processing creates unnecessary bottlenecks. For enterprise datasets with hundreds of files, parallelization can reduce processing time from hours to minutes."
+
+3. **Resource Impact**: "Poor memory management in input processing can cause out-of-memory errors with large datasets. Streaming and chunking are essential for enterprise-scale data processing."
+
+4. **User Experience Impact**: "Input path handling errors are often silent until runtime, causing difficult-to-debug failures. Proper validation and clear error messages save hours of troubleshooting."
+
+5. **Scalability Impact**: "Input processing patterns that work for small datasets can fail catastrophically at enterprise scale. Always design for the largest expected dataset size."
+
+6. **Reliability Impact**: "Input operations are often the first point of failure in data pipelines. Robust error handling and retry logic in input processing prevents entire workflows from failing due to transient issues."

application_sdk/interceptors/.cursor/BUGBOT.md

@@ -0,0 +1,320 @@
+# Interceptor Code Review Guidelines - Temporal Interceptors
+
+## Context-Specific Patterns
+
+This directory contains Temporal interceptor implementations that provide cross-cutting functionality like distributed locking, observability, and event handling. Interceptors must be robust and not interfere with normal workflow/activity execution.
+
+### Phase 1: Critical Interceptor Safety Issues
+
+**Infinite Loop Prevention:**
+
+- **Lock acquisition loops must have termination conditions**: No `while True` loops without max retries or timeouts
+- **Bounded retry logic**: All retry mechanisms must have explicit limits
+- **Timeout enforcement**: Operations must respect activity and workflow timeouts
+- **Resource exhaustion prevention**: Prevent scenarios that could consume all available resources
+
+**Resource Management in Interceptors:**
+
+- **Context manager handling**: Ensure proper cleanup when context managers fail
+- **Connection lifecycle**: Don't hold connections longer than necessary
+- **Lock timing**: Acquire locks as late as possible, release as early as possible
+- **Error state cleanup**: Clean up resources even when intercepted operations fail
+
+```python
+# ✅ DO: Bounded lock acquisition with proper cleanup
+class GoodLockInterceptor:
+    async def intercept_activity(self, next_fn, input):
+        """Intercept with bounded retry and proper cleanup."""
+        max_retries = 10
+        retry_count = 0
+
+        while retry_count < max_retries:  # Bounded loop
+            try:
+                # Acquire lock with timeout
+                async with self.lock_manager.acquire_lock(
+                    lock_name=self.lock_name,
+                    timeout=30  # Maximum lock wait time
+                ) as lock:
+                    # Execute activity while holding lock
+                    return await next_fn(input)
+
+            except LockUnavailableError:
+                retry_count += 1
+                if retry_count >= max_retries:
+                    raise LockAcquisitionError(
+                        f"Failed to acquire lock after {max_retries} attempts"
+                    )
+
+                # Exponential backoff, but bounded
+                delay = min(10 + random.randint(0, 10), 60)  # Max 60 seconds
+                await asyncio.sleep(delay)  # Non-blocking sleep
+
+        raise LockAcquisitionError("Lock acquisition retries exhausted")
+
+# ❌ NEVER: Infinite loops and poor resource management
+class BadLockInterceptor:
+    async def intercept_activity(self, next_fn, input):
+        while True:  # INFINITE LOOP!
+            try:
+                with self.dapr_client.try_lock(lock_name) as response:
+                    if response.success:
+                        result = await next_fn(input)
+                        return result
+                    else:
+                        time.sleep(10)  # BLOCKING SLEEP IN ASYNC!
+                        # SLEEP INSIDE CONTEXT MANAGER - RESOURCE LEAK!
+
+            except Exception as e:
+                raise  # No retry for transient errors
+```
+
+**Parameter Validation and Edge Cases:**
+
+- **Range validation**: Validate parameters that are used in range operations (random.randint, array slicing)
+- **Zero and negative handling**: Explicitly handle edge cases like 0 values, negative numbers
+- **Type validation**: Ensure interceptor parameters match expected types
+- **Configuration validation**: Validate interceptor configuration before use
+
+```python
+# ✅ DO: Comprehensive parameter validation
+def validate_lock_configuration(max_locks: int, ttl_seconds: int) -> None:
+    """Validate lock configuration parameters."""
+
+    if max_locks <= 0:
+        raise ValueError(f"max_locks must be positive, got: {max_locks}")
+
+    if max_locks > 1000:  # Reasonable upper bound
+        raise ValueError(f"max_locks too high (max 1000), got: {max_locks}")
+
+    if ttl_seconds <= 0:
+        raise ValueError(f"ttl_seconds must be positive, got: {ttl_seconds}")
+
+    if ttl_seconds > 3600:  # 1 hour max
+        raise ValueError(f"ttl_seconds too high (max 3600), got: {ttl_seconds}")
+
+def get_random_lock_slot(max_locks: int) -> int:
+    """Get random lock slot with proper validation."""
+    validate_lock_configuration(max_locks, 1)  # Quick validation
+    return random.randint(0, max_locks - 1)  # Safe after validation
+
+# ❌ REJECT: No validation leading to crashes
+def bad_random_slot(max_locks: int) -> int:
+    # Crashes if max_locks is 0 or negative
+    return random.randint(0, max_locks - 1)  # ValueError!
+```
+
+### Phase 2: Interceptor Architecture Patterns
+
+**Lock Acquisition and Release Timing:**
+
+- **Late acquisition**: Acquire locks as close to the protected operation as possible
+- **Early release**: Release locks immediately after the protected operation completes
+- **Timeout alignment**: Lock TTL should align with activity execution time
+- **Race condition prevention**: Ensure locks are held for the entire duration of the protected operation
+
+**Error Handling in Interceptors:**
+
+- **Transient error retry**: Distinguish between retryable errors (connection failures) and permanent errors (invalid configuration)
+- **Interceptor isolation**: Interceptor failures should not prevent other interceptors from running
+- **Activity result preservation**: Don't modify or lose activity results due to interceptor errors
+- **Cleanup on failure**: Ensure resources are cleaned up even when intercepted operations fail
+
+```python
+# ✅ DO: Proper lock timing and error handling
+class ProperLockInterceptor:
+    async def intercept_activity(self, next_fn, input) -> Any:
+        """Intercept with proper timing and error handling."""
+        lock_acquired = False
+        lock_handle = None
+
+        try:
+            # Validate configuration first
+            if self.max_locks <= 0:
+                raise ConfigurationError(f"Invalid max_locks: {self.max_locks}")
+
+            # Bounded retry with exponential backoff
+            for attempt in range(self.max_retries):
+                try:
+                    # Late acquisition - right before protected operation
+                    lock_handle = await self.lock_manager.acquire_lock(
+                        lock_name=self.lock_name,
+                        ttl_seconds=self.ttl_seconds,
+                        timeout=self.lock_timeout
+                    )
+                    lock_acquired = True
+                    break
+
+                except LockUnavailableError:
+                    if attempt >= self.max_retries - 1:
+                        raise LockAcquisitionError(f"Could not acquire lock after {self.max_retries} attempts")
+
+                    # Exponential backoff with jitter
+                    delay = min(2 ** attempt + random.uniform(0, 1), self.max_delay)
+                    await asyncio.sleep(delay)
+                    continue
+
+                except (ConnectionError, TimeoutError) as e:
+                    # Transient errors - retry
+                    logger.warning(f"Transient lock error on attempt {attempt + 1}: {e}")
+                    if attempt >= self.max_retries - 1:
+                        raise LockAcquisitionError(f"Lock service unavailable after {self.max_retries} attempts")
+                    await asyncio.sleep(1)  # Brief delay for transient errors
+                    continue
+
+            # Execute protected operation while holding lock
+            result = await next_fn(input)
+
+            # Early release - immediately after operation
+            if lock_handle:
+                await self.lock_manager.release_lock(lock_handle)
+                lock_acquired = False
+
+            return result
+
+        finally:
+            # Cleanup: ensure lock is released even on failure
+            if lock_acquired and lock_handle:
+                try:
+                    await self.lock_manager.release_lock(lock_handle)
+                except Exception as e:
+                    logger.warning(f"Failed to release lock during cleanup: {e}")
+
+# ❌ REJECT: Poor timing and error handling
+class BadLockInterceptor:
+    async def intercept_activity(self, next_fn, input):
+        # No parameter validation
+        while True:  # Infinite loop
+            lock = await self.acquire_lock()
+            if lock:
+                result = await next_fn(input)
+                # Lock released too early - before result is processed
+                await self.release_lock(lock)
+                return result
+            time.sleep(10)  # Blocking sleep
+```
+
+### Phase 3: Interceptor Testing Requirements
+
+**Interceptor Testing Standards:**
+
+- **Test failure scenarios**: Verify behavior when locks can't be acquired, connections fail, etc.
+- **Test retry logic**: Ensure bounded retries work correctly with various failure patterns
+- **Test resource cleanup**: Verify proper cleanup in both success and failure cases
+- **Test timing**: Ensure locks are acquired/released at correct times
+- **Test integration**: Verify interceptors work correctly with actual activities/workflows
+
+**Edge Case Testing:**
+
+- **Zero and negative parameters**: Test with edge case values like `max_locks=0`
+- **Timeout scenarios**: Test behavior when operations exceed configured timeouts
+- **Concurrent access**: Test interceptor behavior under high concurrency
+- **Resource exhaustion**: Test behavior when external resources are unavailable
+
+### Phase 4: Performance and Scalability
+
+**Interceptor Performance:**
+
+- **Minimal overhead**: Interceptors should add minimal latency to operation execution
+- **Efficient lock management**: Use optimal strategies for lock acquisition and release
+- **Connection pooling**: Reuse connections to external services (Dapr, Redis, etc.)
+- **Async efficiency**: Never block the event loop in async interceptors
+
+**Scalability Patterns:**
+
+- **Bounded resource usage**: Prevent interceptors from consuming unbounded resources
+- **Graceful degradation**: Handle scenarios where external services are unavailable
+- **Circuit breaker patterns**: Implement circuit breakers for external service dependencies
+- **Monitoring and metrics**: Include appropriate metrics for interceptor performance
+
+### Phase 5: Interceptor Maintainability
+
+**Code Organization:**
+
+- **Single responsibility**: Each interceptor should handle one cross-cutting concern
+- **Clear interfaces**: Interceptor interfaces should be well-defined and documented
+- **Configuration externalization**: All interceptor behavior should be configurable
+- **Error reporting**: Provide clear error messages when interceptors fail
+
+**Integration Safety:**
+
+- **Non-interference**: Interceptors should not interfere with each other
+- **Order independence**: Interceptor order should not affect correctness (when possible)
+- **Backwards compatibility**: Changes to interceptors should maintain API compatibility
+- **Graceful failure**: Interceptor failures should not prevent core functionality
+
+---
+
+## Interceptor-Specific Anti-Patterns
+
+**Always Reject:**
+
+- **Infinite retry loops**: `while True` without bounded conditions
+- **Resource leaks in context managers**: Sleeping or blocking inside context managers
+- **Parameter validation gaps**: Not validating inputs that are used in range operations
+- **Blocking operations in async**: Using synchronous operations that block the event loop
+- **Generic error handling**: Not distinguishing between retryable and permanent errors
+- **Lock timing issues**: Releasing locks before operations complete
+- **Missing cleanup**: Not cleaning up resources in failure scenarios
+
+**Lock Acquisition Anti-Patterns:**
+
+```python
+# ❌ REJECT: Multiple critical issues
+class CriticallyFlawedInterceptor:
+    async def intercept(self, next_fn, input):
+        while True:  # 1. Infinite loop
+            async with dapr_client.try_lock(lock_name) as response:
+                if response.success:
+                    return await next_fn(input)
+                else:
+                    time.sleep(10)  # 2. Blocking sleep in async
+                    # 3. Sleep inside context manager - resource leak
+
+        # 4. No parameter validation for max_locks
+        slot = random.randint(0, max_locks - 1)  # Crashes if max_locks <= 0
+
+# ✅ REQUIRE: Proper implementation
+class WellImplementedInterceptor:
+    def __init__(self, max_locks: int = 10, max_retries: int = 5):
+        # Validate configuration at initialization
+        if max_locks <= 0:
+            raise ValueError(f"max_locks must be positive: {max_locks}")
+        self.max_locks = max_locks
+        self.max_retries = max_retries
+
+    async def intercept(self, next_fn, input) -> Any:
+        for attempt in range(self.max_retries):  # Bounded retries
+            try:
+                # Context manager properly exits before sleep
+                async with self.lock_client.try_lock(self.lock_name) as response:
+                    if response.success:
+                        return await next_fn(input)
+
+                # Sleep outside context manager
+                if attempt < self.max_retries - 1:
+                    await asyncio.sleep(min(2 ** attempt, 30))  # Non-blocking sleep
+
+            except ConnectionError:
+                # Retry transient errors
+                if attempt >= self.max_retries - 1:
+                    raise
+                await asyncio.sleep(1)
+
+        raise LockAcquisitionError("Could not acquire lock within retry limit")
+```
+
+## Educational Context for Interceptor Reviews
+
+When reviewing interceptor code, emphasize:
+
+1. **System Stability Impact**: "Interceptors run for every activity/workflow execution. Infinite loops or resource leaks in interceptors can bring down the entire system by exhausting resources."
+
+2. **Performance Impact**: "Interceptor overhead affects every operation. Blocking operations in async interceptors can degrade performance for all concurrent executions."
+
+3. **Reliability Impact**: "Poor error handling in interceptors can mask or cause cascading failures. Proper error distinction and recovery logic are essential for system reliability."
+
+4. **Resource Impact**: "Interceptors often manage external resources (locks, connections). Resource leaks in interceptors compound quickly under load and can cause system-wide failures."
+
+5. **Debugging Impact**: "Interceptor issues are often hard to debug because they affect multiple operations. Clear error messages and proper logging are critical for troubleshooting."
+
+6. **Scalability Impact**: "Interceptor patterns that work under light load can fail catastrophically under heavy load. Always design for high-concurrency scenarios with proper resource bounds."