atlan-application-sdk 0.1.1rc39__py3-none-any.whl → 0.1.1rc40__py3-none-any.whl

application_sdk/activities/.cursor/BUGBOT.md

@@ -0,0 +1,424 @@
+# Activity Code Review Guidelines - Temporal Activities
+
+## Context-Specific Patterns
+
+This directory contains Temporal activity implementations that perform the actual work of workflows. Activities handle external I/O, database operations, and non-deterministic tasks.
+
+### Phase 1: Critical Activity Safety Issues
+
+**External Resource Safety:**
+
+- All external connections (database, API, file) must have explicit timeouts
+- Connection failures must be handled gracefully with proper retry logic
+- Resource cleanup must happen in finally blocks or context managers
+- Sensitive data must not be logged or exposed in error messages
+- All user inputs must be validated before processing
+
+**Activity Timeout Management:**
+
+- Activities must respect Temporal heartbeat timeouts for long-running operations
+- Progress should be reported via heartbeat for operations > 30 seconds
+- Activities should check for cancellation requests periodically
+- Timeout values must be realistic for the operation being performed
+
+```python
+# ✅ DO: Proper activity with heartbeat and cancellation
+@activity.defn
+async def process_large_dataset_activity(dataset_config: dict) -> dict:
+    total_records = await get_record_count(dataset_config)
+    processed = 0
+
+    async for batch in process_in_batches(dataset_config):
+        # Check for cancellation
+        activity.heartbeat({"progress": processed, "total": total_records})
+
+        try:
+            await process_batch(batch)
+            processed += len(batch)
+        except Exception as e:
+            activity.logger.error(f"Batch processing failed: {e}", exc_info=True)
+            raise
+
+    return {"processed_records": processed}
+
+# ❌ NEVER: Long-running activity without heartbeat
+@activity.defn
+async def bad_process_activity(data):
+    # No heartbeat, no cancellation check, no progress reporting
+    return await process_all_data_at_once(data)
+```
+
+### Phase 2: Activity Architecture Patterns
+
+**Resource Management:**
+
+- Use connection pooling for database operations
+- Implement proper connection context managers
+- Clean up temporary files and resources
+- Handle partial failures gracefully
+- Implement idempotent operations where possible
+
+**Default Value Management:**
+
+- **Always define sensible defaults**: Activity parameters should have reasonable default values where appropriate
+- **Avoid required parameters for inferable values**: Values like `owner_id` that can be derived (e.g., from `application_name:run_id`) should not be required parameters
+- **Default TTL values**: Lock operations, cache entries, and timeouts should have documented default values (e.g., 300 seconds for locks)
+- **Environment-based defaults**: Different environments (dev/prod) may need different defaults
+
+```python
+# ✅ DO: Proper default value management
+@activity.defn
+async def acquire_distributed_lock_activity(
+    lock_name: str,
+    max_locks: int = 10,  # Sensible default
+    ttl_seconds: int = 300,  # 5 minutes default
+    owner_id: Optional[str] = None  # Will be inferred
+) -> dict:
+    """Acquire a distributed lock with proper defaults."""
+
+    # Infer owner_id if not provided
+    if owner_id is None:
+        workflow_info = activity.info().workflow_execution
+        owner_id = f"{workflow_info.workflow_type}:{workflow_info.workflow_id}"
+
+    # Validate parameters
+    if max_locks <= 0:
+        raise ValueError(f"max_locks must be positive, got: {max_locks}")
+
+    return await lock_manager.acquire_lock(lock_name, max_locks, ttl_seconds, owner_id)
+
+# ❌ REJECT: Poor parameter management
+@activity.defn
+async def bad_acquire_lock_activity(
+    lock_name: str,
+    max_locks: int,  # No default
+    ttl_seconds: int,  # No default
+    owner_id: str,  # Required but could be inferred
+    application_name: str,  # Redundant - should be inferred
+    run_id: str  # Redundant - should be inferred
+) -> dict:
+    # Forces users to pass values that could be automatically determined
+    pass
+```
+
+**Error Handling and Retries:**
+
+- Distinguish between retryable and non-retryable errors
+- Use specific exception types for different error conditions
+- Log errors with sufficient context for debugging
+- Implement exponential backoff for retryable operations
+- Preserve error context across retries
+
+```python
+# ✅ DO: Proper error handling with context
+@activity.defn
+async def extract_metadata_activity(connection_config: dict) -> dict:
+    client = None
+    try:
+        client = await create_database_client(connection_config)
+        await client.validate_connection()
+
+        metadata = await client.extract_metadata()
+
+        activity.logger.info(
+            f"Extracted metadata for {len(metadata)} objects",
+            extra={"database": connection_config.get("database", "unknown")}
+        )
+
+        return metadata
+
+    except ConnectionError as e:
+        # Retryable error
+        activity.logger.warning(f"Connection failed, will retry: {e}")
+        raise  # Let Temporal handle retry
+
+    except ValidationError as e:
+        # Non-retryable error
+        activity.logger.error(f"Invalid connection config: {e}")
+        raise ApplicationError(f"Configuration validation failed: {e}", non_retryable=True)
+
+    finally:
+        if client:
+            await client.close()
+```
+
+**Resource Validation and Limits:**
+
+- **Key length validation**: Ensure generated keys (Redis, cache) don't exceed system limits
+- **Memory constraints**: Validate that operations won't exceed available memory
+- **Connection limits**: Check that concurrent operations stay within connection pool limits
+- **Processing time estimates**: Validate that operations can complete within activity timeouts
+
+```python
+# ✅ DO: Resource validation
+@activity.defn
+async def process_with_validation_activity(
+    resource_name: str,
+    data_size_mb: int,
+    max_processing_time_minutes: int = 30
+) -> dict:
+    """Process data with proper resource validation."""
+
+    # Validate resource constraints
+    if len(resource_name.encode('utf-8')) > 512 * 1024 * 1024:  # 512MB Redis key limit
+        raise ValueError(f"Resource name too long: {len(resource_name)} bytes")
+
+    if data_size_mb > 1000:  # 1GB memory limit
+        raise ValueError(f"Data size {data_size_mb}MB exceeds 1GB limit")
+
+    # Validate processing time against activity timeout
+    activity_timeout = activity.info().start_to_close_timeout
+    if max_processing_time_minutes * 60 > activity_timeout.total_seconds():
+        raise ValueError(f"Processing time {max_processing_time_minutes}m exceeds timeout")
+
+    return await process_data(resource_name, data_size_mb)
+```
+
+### Phase 3: Activity Testing Requirements
+
+**Activity Testing Standards:**
+
+- Test activities independently from workflows
+- Mock external dependencies (databases, APIs, file systems)
+- Test timeout and cancellation behaviors
+- Test retry scenarios with different error types
+- Include performance tests for long-running activities
+- Test heartbeat and progress reporting
+
+**Integration Testing:**
+
+- Use test databases/services for integration tests
+- Test real connection failures and recovery
+- Verify proper resource cleanup
+- Test activity behavior under load
+- Include end-to-end tests with real workflows
+
+### Phase 4: Performance and Scalability
+
+**Activity Performance:**
+
+- Use async/await for all I/O operations
+- Implement proper batching for bulk operations
+- Use streaming for large datasets
+- Monitor activity execution time and resource usage
+- Optimize database queries and API calls
+
+**Memory Management:**
+
+- Process large datasets in chunks, not all at once
+- Use generators for memory-efficient iteration
+- Clean up large objects explicitly
+- Monitor memory usage in long-running activities
+- Use appropriate data types and structures
+
+```python
+# ✅ DO: Memory-efficient processing
+@activity.defn
+async def process_large_file_activity(file_path: str, chunk_size: int = 1000) -> dict:
+    processed_count = 0
+
+    async with aiofiles.open(file_path, 'r') as file:
+        chunk = []
+        async for line in file:
+            chunk.append(line.strip())
+
+            if len(chunk) >= chunk_size:
+                await process_chunk(chunk)
+                processed_count += len(chunk)
+                chunk = []
+
+                # Report progress and check for cancellation
+                activity.heartbeat({"processed": processed_count})
+
+        # Process remaining items
+        if chunk:
+            await process_chunk(chunk)
+            processed_count += len(chunk)
+
+    return {"total_processed": processed_count}
+
+# ❌ NEVER: Load entire file into memory
+@activity.defn
+async def bad_file_activity(file_path: str):
+    with open(file_path, 'r') as file:
+        all_lines = file.readlines()  # Memory intensive!
+    return process_all_lines(all_lines)
+```
+
+**Parallelization Opportunities:**
+
+- **Flag sequential operations**: When processing multiple files or resources, suggest parallel processing
+- **Batch operations**: Group related operations to reduce overhead
+- **Connection reuse**: Optimize connection usage across operations
+- **Async patterns**: Ensure I/O operations don't block other processing
+
+### Phase 5: Activity Maintainability
+
+**Code Organization:**
+
+- Keep activities focused on a single responsibility
+- Use dependency injection for external services
+- Implement proper logging with activity context
+- Document activity parameters and return values
+- Follow consistent naming conventions
+
+**Configuration and Environment:**
+
+- Externalize all configuration parameters
+- Use environment-specific settings appropriately
+- Validate configuration before using it
+- Support development and production configurations
+- Document all required configuration options
+
+**Error Context Enhancement:**
+
+- **Operation identification**: Include the specific operation that failed in error messages
+- **Parameter context**: Log relevant parameters (sanitized) when operations fail
+- **Resource state**: Include information about resource availability/state in errors
+- **Recovery suggestions**: Where possible, include suggestions for resolving errors
+
+```python
+# ✅ DO: Enhanced error context
+@activity.defn
+async def enhanced_error_activity(
+    database_name: str,
+    table_names: List[str],
+    timeout_seconds: int = 300
+) -> dict:
+    """Activity with comprehensive error context."""
+
+    try:
+        result = await extract_table_metadata(database_name, table_names, timeout_seconds)
+        return result
+
+    except ConnectionTimeout as e:
+        activity.logger.error(
+            f"Database connection timeout during metadata extraction",
+            extra={
+                "database": database_name,
+                "tables_requested": len(table_names),
+                "timeout_used": timeout_seconds,
+                "suggestion": "Consider increasing timeout or reducing table count"
+            }
+        )
+        raise ApplicationError(
+            f"Metadata extraction timed out after {timeout_seconds}s for database '{database_name}' "
+            f"with {len(table_names)} tables. Consider reducing scope or increasing timeout.",
+            non_retryable=True
+        )
+
+    except InsufficientPrivileges as e:
+        activity.logger.error(
+            f"Insufficient database privileges for metadata extraction",
+            extra={
+                "database": database_name,
+                "required_privileges": ["SELECT", "INFORMATION_SCHEMA_READ"],
+                "suggestion": "Grant required database privileges to connection user"
+            }
+        )
+        raise ApplicationError(
+            f"Missing database privileges for '{database_name}'. "
+            f"Ensure connection user has SELECT and INFORMATION_SCHEMA access.",
+            non_retryable=True
+        )
+```
+
+---
+
+## Activity-Specific Anti-Patterns
+
+**Always Reject:**
+
+- Activities without proper timeout handling
+- Long-running activities without heartbeat reporting
+- Missing resource cleanup (connections, files, etc.)
+- Generic exception handling without specific error types
+- Activities that don't handle cancellation
+- Synchronous I/O operations in async activities
+- Missing logging for error conditions
+- Activities without proper input validation
+
+**Parameter Management Anti-Patterns:**
+
+- **Over-parameterization**: Requiring parameters that can be inferred from context
+- **Missing defaults**: Parameters without reasonable default values
+- **No validation**: Accepting parameters without validating constraints
+- **Redundant parameters**: Multiple parameters representing the same concept
+
+**Resource Management Anti-Patterns:**
+
+```python
+# ❌ REJECT: Poor resource management
+@activity.defn
+async def bad_database_activity(query: str):
+    # No connection pooling, no cleanup, no error handling
+    conn = await psycopg.connect("host=localhost...")
+    result = await conn.execute(query)  # No timeout
+    return result.fetchall()  # Connection never closed
+
+# ✅ REQUIRE: Proper resource management
+@activity.defn
+async def good_database_activity(query: str, params: tuple = ()) -> list:
+    async with get_connection_pool().acquire() as conn:
+        try:
+            # Set query timeout
+            async with conn.cursor() as cursor:
+                await cursor.execute(query, params)
+                return await cursor.fetchall()
+        except Exception as e:
+            activity.logger.error(f"Database query failed: {query[:100]}...", exc_info=True)
+            raise
+        # Connection automatically returned to pool
+```
+
+**Heartbeat and Cancellation Anti-Patterns:**
+
+```python
+# ❌ REJECT: No heartbeat or cancellation handling
+@activity.defn
+async def bad_long_running_activity(data_list: list):
+    results = []
+    for item in data_list:  # Could take hours
+        result = await expensive_operation(item)
+        results.append(result)
+    return results
+
+# ✅ REQUIRE: Proper heartbeat and cancellation
+@activity.defn
+async def good_long_running_activity(data_list: list) -> list:
+    results = []
+    total_items = len(data_list)
+
+    for i, item in enumerate(data_list):
+        # Check for cancellation and report progress
+        activity.heartbeat({
+            "processed": i,
+            "total": total_items,
+            "percent_complete": (i / total_items) * 100
+        })
+
+        try:
+            result = await expensive_operation(item)
+            results.append(result)
+        except Exception as e:
+            activity.logger.error(f"Processing failed for item {i}: {e}")
+            raise
+
+    return results
+```
+
+## Educational Context for Activity Reviews
+
+When reviewing activity code, emphasize:
+
+1. **Reliability Impact**: "Activities are where the real work happens. Proper error handling and resource management in activities determines whether workflows succeed or fail under real-world conditions."
+
+2. **Performance Impact**: "Activity performance directly affects workflow execution time. Inefficient activities create bottlenecks that slow down entire business processes."
+
+3. **Observability Impact**: "Activity logging and heartbeat reporting are essential for monitoring long-running processes. Without proper observability, debugging workflow issues becomes nearly impossible."
+
+4. **Resource Impact**: "Activities consume actual system resources. Poor resource management in activities can cause memory leaks, connection pool exhaustion, and system instability."
+
+5. **Cancellation Impact**: "Activities that don't handle cancellation properly can continue consuming resources even after workflows are cancelled, leading to resource waste and potential system overload."
+
+6. **Parameter Design Impact**: "Well-designed activity parameters with sensible defaults make activities easier to use and less error-prone. Over-parameterization creates maintenance burden and increases the chance of configuration errors."

application_sdk/clients/.cursor/BUGBOT.md

@@ -0,0 +1,280 @@
+# Client Code Review Guidelines - Database and External Services
+
+## Context-Specific Patterns
+
+This directory contains database clients, external service clients, and connection management code. These components are critical for data integrity, performance, and security.
+
+### Phase 1: Critical Client Safety Issues
+
+**Database Connection Security:**
+
+- SQL injection prevention through parameterized queries ONLY
+- Connection strings must never contain hardcoded credentials
+- Database passwords must be retrieved from secure credential stores
+- SSL/TLS required for all external database connections
+- Connection timeouts must be explicitly configured
+
+**Example SQL Injection Prevention:**
+
+```python
+# ✅ DO: Parameterized queries
+cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))
+
+# ❌ NEVER: String concatenation
+cursor.execute(f"SELECT * FROM users WHERE id = {user_id}")
+```
+
+### Phase 2: Client Architecture Patterns
+
+**Connection Pooling Requirements:**
+
+- All database clients MUST use connection pooling
+- Pool size must be configurable via environment variables
+- Connection validation on checkout required
+- Proper connection cleanup in finally blocks
+- Connection leak detection in development/testing
+
+**Class Responsibility Separation:**
+
+- **Always flag multi-responsibility classes**: Classes handling both client functionality and domain-specific logic must be separated
+- **Client vs Logic separation**: Database clients should handle connections, not business rules
+- **Extract domain logic**: Lock management, caching, or processing logic should be in separate classes
+- **Single purpose interfaces**: Each client class should have one clear responsibility
+
+```python
+# ❌ REJECT: Mixed responsibilities
+class RedisClient:
+    def connect(self):
+        """Handle connection setup"""
+    def acquire_lock(self, lock_name):
+        """Lock functionality - should be separate"""
+    def get_data(self, key):
+        """Client functionality"""
+
+# ✅ REQUIRE: Separated responsibilities
+class RedisClient:
+    def connect(self):
+        """Handle connection setup"""
+    def get_data(self, key):
+        """Client functionality"""
+
+class RedisLockManager:  # Separate class for lock functionality
+    def __init__(self, client: RedisClient):
+        self.client = client
+    def acquire_lock(self, lock_name):
+        """Lock-specific logic"""
+```
+
+**Async Client Patterns:**
+
+- Use async/await for all I/O operations
+- Implement proper connection context managers
+- Handle connection failures gracefully with retries
+- Use asyncio connection pools, not synchronous pools
+
+```python
+# ✅ DO: Proper async connection management
+async def execute_query(self, query: str, params: tuple):
+    async with self.pool.acquire() as conn:
+        try:
+            return await conn.fetch(query, *params)
+        except Exception as e:
+            logger.error(f"Query failed: {query[:100]}...", exc_info=True)
+            raise
+```
+
+**Configuration Management for Clients:**
+
+- **Environment-specific settings**: All connection parameters must be externalized to environment variables
+- **Default value validation**: Every configuration parameter must have a sensible default and validation
+- **Development vs Production**: Client configurations must work in both environments
+- **Configuration consolidation**: Related configuration should be grouped together
+
+```python
+# ✅ DO: Proper client configuration
+class DatabaseClientConfig:
+    def __init__(self):
+        self.host = os.getenv("DB_HOST", "localhost")
+        self.port = int(os.getenv("DB_PORT", "5432"))
+        self.max_connections = int(os.getenv("DB_MAX_CONNECTIONS", "20"))
+        self.timeout = int(os.getenv("DB_TIMEOUT_SECONDS", "30"))
+        self.ssl_required = os.getenv("DB_SSL_REQUIRED", "true").lower() == "true"
+        self._validate()
+
+    def _validate(self):
+        if self.max_connections <= 0:
+            raise ValueError("DB_MAX_CONNECTIONS must be positive")
+        if self.timeout <= 0:
+            raise ValueError("DB_TIMEOUT_SECONDS must be positive")
+
+# ❌ REJECT: Poor configuration management
+class BadDatabaseClient:
+    def __init__(self):
+        self.host = "localhost"  # Hardcoded
+        self.connections = os.getenv("MAX_CONN")  # No default, no validation
+```
+
+### Phase 3: Client Testing Requirements
+
+**Database Client Testing:**
+
+- Mock database connections in unit tests
+- Use test databases for integration tests
+- Test connection failure scenarios
+- Verify connection pool behavior
+- Test query parameter sanitization
+- Include performance tests for connection pooling
+
+**External Service Client Testing:**
+
+- Mock external APIs in unit tests
+- Test timeout and retry behaviors
+- Test authentication failure scenarios
+- Include circuit breaker tests
+- Verify proper error handling and logging
+
+### Phase 4: Performance and Scalability
+
+**Query Performance:**
+
+- Flag SELECT \* queries without LIMIT
+- Require WHERE clauses on indexed columns
+- Batch operations when possible
+- Use prepared statements for repeated queries
+- Monitor and limit query execution time
+
+**Connection Management Performance:**
+
+- Connection pool size must match expected concurrency
+- Connection validation queries must be lightweight
+- Implement connection health checks
+- Use connection keepalive for long-running connections
+- Monitor connection pool metrics
+
+**Resource Limit Validation:**
+
+- **Key length constraints**: Validate Redis key lengths against limits (typically 512MB max)
+- **Connection limits**: Ensure connection pool sizes don't exceed database limits
+- **Query complexity**: Monitor and limit expensive query execution time
+- **Memory constraints**: Validate result set sizes for large queries
+
+```python
+# ✅ DO: Resource validation
+def validate_redis_key(key: str) -> str:
+    if len(key.encode('utf-8')) > 512 * 1024 * 1024:  # 512MB limit
+        raise ValueError(f"Redis key too long: {len(key)} bytes")
+    return key
+
+def create_lock_key(application: str, resource: str, run_id: str) -> str:
+    key = f"{application}:{resource}:{run_id}"
+    return validate_redis_key(key)
+```
+
+### Phase 5: Client Maintainability
+
+**Code Organization:**
+
+- Separate client interface from implementation
+- Use dependency injection for client configuration
+- Implement proper logging with connection context
+- Document connection parameters and requirements
+- Follow consistent error handling patterns
+
+**Error Handling Improvements:**
+
+- **Comprehensive try-catch blocks**: All client operations that can fail must be wrapped in try-catch blocks
+- **SDK-specific exceptions**: Use `ClientError` from `application_sdk/common/error_codes.py` instead of generic exceptions
+- **Operation context**: Include operation details (query, connection info) in error messages
+- **Retry vs fail-fast**: Distinguish between retryable connection errors and permanent failures
+
+```python
+# ✅ DO: Comprehensive error handling
+from application_sdk.common.error_codes import ClientError
+
+async def execute_query(self, query: str, params: tuple = ()) -> list:
+    try:
+        async with self.pool.acquire() as conn:
+            return await conn.fetch(query, *params)
+    except ConnectionRefusedError as e:
+        # Retryable error
+        logger.warning(f"Database connection refused, will retry: {e}")
+        raise ClientError(f"Database temporarily unavailable: {e}")
+    except ValidationError as e:
+        # Non-retryable error
+        logger.error(f"Query validation failed: {query[:50]}...")
+        raise ClientError(f"Invalid query: {e}")
+    except Exception as e:
+        logger.error(f"Unexpected database error: {e}", exc_info=True)
+        raise ClientError(f"Database operation failed: {e}")
+```
+
+**Configuration Management:**
+
+- Externalize all connection parameters
+- Support multiple environment configurations
+- Implement configuration validation
+- Use secure credential management
+- Document all configuration options
+
+---
+
+## Client-Specific Anti-Patterns
+
+**Always Reject:**
+
+- Hardcoded connection strings or credentials
+- Missing connection timeouts
+- Synchronous database calls in async contexts
+- SQL queries built through string concatenation
+- Connection objects stored as instance variables
+- Missing connection pool cleanup
+- Generic exception handling without context
+- Direct database connections without pooling
+
+**Configuration Anti-Patterns:**
+
+- **Missing environment variables**: Parameters that should be configurable but are hardcoded
+- **No validation**: Environment variables used without type checking or range validation
+- **Missing defaults**: Required configuration without sensible fallback values
+- **Environment inconsistency**: Features that work in development but fail in production
+
+**Connection Management Anti-Patterns:**
+
+```python
+# ❌ REJECT: Poor connection management
+class BadSQLClient:
+    def __init__(self):
+        self.conn = psycopg2.connect("host=localhost...")  # No pooling
+
+    def query(self, sql):
+        cursor = self.conn.cursor()
+        cursor.execute(sql)  # No parameterization
+        return cursor.fetchall()  # No cleanup
+
+# ✅ REQUIRE: Proper connection management
+class GoodSQLClient:
+    def __init__(self, pool: ConnectionPool):
+        self.pool = pool
+
+    async def query(self, sql: str, params: tuple = ()):
+        async with self.pool.acquire() as conn:
+            try:
+                return await conn.fetch(sql, *params)
+            finally:
+                # Connection automatically returned to pool
+                pass
+```
+
+## Educational Context for Client Reviews
+
+When reviewing client code, emphasize:
+
+1. **Security Impact**: "Database clients are the primary attack vector for SQL injection. Parameterized queries aren't just best practice - they're essential for protecting enterprise customer data."
+
+2. **Performance Impact**: "Connection pooling isn't optional at enterprise scale. Creating new connections for each query can overwhelm database servers and create bottlenecks that affect all users."
+
+3. **Reliability Impact**: "Proper error handling in clients determines whether temporary network issues cause cascading failures or graceful degradation."
+
+4. **Maintainability Impact**: "Client abstraction layers allow us to change databases or connection strategies without affecting business logic throughout the application."
+
+5. **Configuration Impact**: "Externalized configuration enables the same code to work across development, staging, and production environments. Missing this leads to environment-specific bugs that are hard to reproduce and fix."