PECLibrary 0.7.0__tar.gz

peclibrary-0.7.0/ARCHITECTURE.md

@@ -0,0 +1,484 @@
+# Palantir Event Capture - Architecture
+
+## Overview
+
+The `palantir-event-capture` library provides a comprehensive event capture system for data pipelines. It follows a layered architecture with clear separation of concerns.
+
+## Architecture Layers
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Application Layer                         │
+│              (User Functions/Pipelines)                      │
+└───────────────────┬─────────────────────────────────────────┘
+                    │
+                    │ @capture_errors
+                    │ @async_capture_errors
+                    │ @retry_with_capture
+                    │
+┌───────────────────▼─────────────────────────────────────────┐
+│                   Decorator Layer                            │
+│          (decorators/capture.py)                             │
+│  - Function wrapping                                         │
+│  - Exception handling                                        │
+│  - Context extraction                                        │
+└───────────────────┬─────────────────────────────────────────┘
+                    │
+                    │ ErrorEvent
+                    │
+┌───────────────────▼─────────────────────────────────────────┐
+│                  Service Layer                               │
+│          (services/error_capture.py)                         │
+│  - Event validation                                          │
+│  - Metadata enrichment                                       │
+│  - Transport routing                                         │
+└───────────────────┬─────────────────────────────────────────┘
+                    │
+            ┌───────┴───────┐
+            │               │
+┌───────────▼────┐  ┌──────▼──────────┐
+│  Storage Layer │  │ Transport Layer │
+│   (Optional)   │  │ (Kafka/Custom)  │
+│  - Local cache │  │ - Kafka         │
+│  - Database    │  │ - Console       │
+└────────────────┘  └─────────────────┘
+```
+
+## Core Components
+
+### 1. Models Layer (`models/`)
+
+**Purpose**: Define data structures and validation schemas
+
+- `captured_models.py`: Pydantic models for events and configuration
+  - `ErrorEvent`: Standardized event structure
+  - `CaptureConfig`: Configuration for capture behavior
+  - `PipelineComponent`: Enum of supported components (expanded list)
+  - `Severity`: Event severity levels
+  - `EventType`: Types of events (ERROR, WARNING, INFO, SUCCESS)
+
+**Key Features**:
+
+- Pydantic validation (V1 and V2 compatible)
+- UUID tracking prevents duplicate events
+- Timezone-aware timestamps (UTC)
+- JSON serialization support
+
+### 2. Decorators Layer (`decorators/`)
+
+**Purpose**: Provide easy-to-use function wrappers
+
+- `capture.py`: Core decorator implementations
+  - `@capture_errors`: Sync function error capture
+  - `@async_capture_errors`: Async function error capture
+  - `@retry_with_capture`: Retry logic with capture
+  - `@capture_method_errors`: Class-level decoration
+
+**Responsibilities**:
+
+- Wrap user functions transparently
+- Extract execution context (function name, file, line number)
+- Handle exceptions and capture stack traces
+- Pass events to service layer
+
+### 3. Services Layer (`services/`)
+
+**Purpose**: Core business logic and coordination
+
+- `error_capture.py`: Main capture service
+
+  - Event creation and validation
+  - Frame analysis for context extraction
+  - Transport coordination
+  - Singleton pattern for global access
+- `transport.py`: Event transport abstraction
+
+  - Base `Transport` interface
+  - `KafkaTransport`: Kafka producer wrapper
+  - `ConsoleTransport`: Console logging
+  - `MultiTransport`: Combine multiple transports
+  - `BackgroundTransport`: Async background worker for any transport
+- `storage_service.py`: Optional event persistence
+
+  - Local file storage
+  - Database integration
+  - Event query and retrieval
+
+**Key Design Patterns**:
+
+- Singleton: Global service instance
+- Strategy: Pluggable transport backends
+- Chain of Responsibility: Multiple transports
+
+### 4. Configuration Layer (`config/`)
+
+**Purpose**: Centralized configuration management
+
+- `defaults.py`: Default configuration values
+- `settings.py`: Configuration loading and validation
+  - Environment variable support
+  - Programmatic configuration
+  - Config validation
+
+**Configuration Sources** (priority order):
+
+1. Programmatic configuration
+2. Environment variables
+3. Default values
+
+### 5. Utilities Layer (`utils/`)
+
+**Purpose**: Reusable helper functions
+
+- `helpers.py`: Common utility functions
+
+  - Frame analysis
+  - Stack trace formatting
+  - Type conversions
+- `context_managers.py`: Context manager implementations
+
+  - `capture_context`: Capture errors in code blocks
+  - Automatic cleanup and error propagation
+- `logging.py`: Logging setup and configuration
+
+  - `structured_logging`: Loguru integration
+  - `monitor_errors.py`: Simple Kafka event monitoring script
+
+## Data Flow
+
+### 1. Normal Execution Flow
+
+```
+User Function
+     │
+     │ Decorated with @capture_errors
+     │
+     ▼
+Decorator intercepts execution
+     │
+     │ try: execute function
+     │
+     ▼
+Function executes successfully
+     │
+     ▼
+Return result to user
+```
+
+### 2. Error Capture Flow
+
+```
+User Function raises Exception
+     │
+     ▼
+Decorator catches exception
+     │
+     ├─► Extract frame info (file, line, function)
+     ├─► Extract stack trace
+     └─► Create CaptureConfig
+     │
+     ▼
+ErrorCaptureService.capture_exception()
+     │
+     ├─► Validate exception type
+     ├─► Enrich metadata
+     ├─► Create ErrorEvent (Pydantic)
+     │
+     ▼
+For each registered transport:
+     │
+     ├─► ConsoleTransport.send()
+     │    └─► Log to console
+     │
+     ├─► KafkaTransport.send()
+     │    └─► Producer.send(topic, event)
+     │
+     └─► CustomTransport.send()
+          └─► Your custom logic
+     │
+     ▼
+Reraise exception (if configured)
+```
+
+### 3. Retry Flow
+
+```
+User Function (with @retry_with_capture)
+     │
+     ▼
+Attempt 1: Exception raised
+     │
+     ├─► Capture event (LOW severity)
+     ├─► Sleep delay (e.g., 1s)
+     │
+     ▼
+Attempt 2: Exception raised
+     │
+     ├─► Capture event (LOW severity)
+     ├─► Sleep delay * backoff (e.g., 2s)
+     │
+     ▼
+Attempt 3 (final): Exception raised
+     │
+     ├─► Capture event (HIGH severity)
+     ├─► Send to transport
+     │
+     ▼
+Raise exception to caller
+```
+
+## Event Model Schema
+
+```python
+ErrorEvent:
+  id: UUID                          # Unique event identifier
+  event_type: EventType             # ERROR, WARNING, INFO, SUCCESS
+  
+  # Pipeline Context
+  pipeline_name: Optional[str]      # Name of the pipeline
+  engine_name: Optional[str]        # Processing engine
+  
+  # Component Info
+  component: PipelineComponent      # KAFKA, POSTGRES, etc.
+  component_name: Optional[str]     # Custom component name
+  severity: Severity                # CRITICAL, HIGH, MEDIUM, LOW
+  
+  # Error Details
+  message: str                      # Human-readable message
+  timestamp: float                  # Unix timestamp
+  timestamp_iso: str                # ISO 8601 format
+  error_type: str                   # Exception class name
+  stack_trace: Optional[str]        # Full stack trace
+  
+  # Execution Context
+  function_name: Optional[str]      # Function where error occurred
+  file_path: Optional[str]          # File path
+  line_number: Optional[int]        # Line number
+  metadata: Dict[str, Any]          # Custom metadata
+  
+  # Resolution Tracking
+  resolved: bool                    # Resolution status
+  resolved_at: Optional[float]      # Resolution timestamp
+  resolved_by: Optional[str]        # Who resolved it
+  resolution_notes: Optional[str]   # Resolution notes
+```
+
+## Transport Architecture
+
+### Base Transport Interface
+
+```python
+class Transport(ABC):
+    @abstractmethod
+    def send(self, event: ErrorEvent) -> bool:
+        """Send event to destination"""
+        pass
+  
+    @abstractmethod
+    def close(self) -> None:
+        """Cleanup resources"""
+        pass
+```
+
+### Kafka Transport
+
+```python
+KafkaTransport:
+  - Lazy connection (on first send)
+  - Automatic reconnection
+  - Topic routing by severity
+  - Batch sending support
+  - Compression (optional)
+```
+
+### Multi Transport
+
+Combine multiple transports:
+
+```python
+MultiTransport([
+    ConsoleTransport(),
+    BackgroundTransport(KafkaTransport()),
+    CustomDatabaseTransport()
+])
+```
+
+### Background Transport
+
+Wraps any other transport to send events asynchronously using a worker thread and queue:
+
+```python
+transport = BackgroundTransport(
+    base_transport=KafkaTransport(),
+    max_queue_size=1000
+)
+```
+
+## Configuration System
+
+### Environment Variables
+
+```bash
+# Kafka
+KAFKA_BOOTSTRAP_SERVERS_RCA=localhost:9092
+KAFKA_EVENT_TOPIC_RCA=rca.events
+KAFKA_ENABLED_RCA=true
+KAFKA_COMPRESSION_RCA=gzip
+
+# Behavior
+LOG_LEVEL=INFO
+LOG_TO_CONSOLE=true
+SEND_TO_TRANSPORT=true
+DEFAULT_SEVERITY=MEDIUM
+ENABLE_STACK_TRACES=true
+USE_BACKGROUND_TRANSPORT=false
+
+# Retry
+DEFAULT_MAX_RETRIES=3
+DEFAULT_RETRY_DELAY=1.0
+DEFAULT_RETRY_BACKOFF=2.0
+```
+
+### Configuration Priority
+
+1. **Decorator arguments** (highest priority)
+2. **Programmatic configuration** via `configure()`
+3. **Environment variables**
+4. **Default values** (lowest priority)
+
+## Error Handling Philosophy
+
+### 1. Fail-Safe Design
+
+The library never causes application failures:
+
+- Capture errors are logged, not raised
+- Transport failures are silent
+- Invalid configuration falls back to defaults
+
+### 2. Zero Performance Impact
+
+- Minimal overhead on success path
+- Lazy initialization
+- Async transport (optional)
+- Batch sending (optional)
+
+### 3. Rich Context
+
+Capture maximum context without user effort:
+
+- Automatic stack traces
+- Function metadata
+- Frame analysis
+- Custom metadata support
+
+## Extension Points
+
+### 1. Custom Transport
+
+```python
+class MyTransport(Transport):
+    def send(self, event: ErrorEvent) -> bool:
+        # Your logic here
+        return True
+  
+    def close(self) -> None:
+        # Cleanup
+        pass
+```
+
+### 2. Custom Storage
+
+```python
+class MyStorage(StorageService):
+    def save(self, event: ErrorEvent) -> None:
+        # Your storage logic
+        pass
+  
+    def query(self, filters: dict) -> List[ErrorEvent]:
+        # Your query logic
+        pass
+```
+
+### 3. Event Enrichment
+
+```python
+def enrich_event(event: ErrorEvent) -> ErrorEvent:
+    event.metadata['hostname'] = socket.gethostname()
+    event.metadata['env'] = os.getenv('ENV')
+    return event
+
+service = get_capture_service()
+service.add_enricher(enrich_event)
+```
+
+## Performance Considerations
+
+### Memory
+
+- Events are created on-demand
+- UUID registry cleaned periodically
+- Stack traces truncated (configurable)
+- Metadata size limits (recommended)
+
+### Latency
+
+- Decorator overhead: ~0.1ms (no error)
+- Event creation: ~0.5ms
+- Kafka send (async): ~1-5ms
+- Synchronous send: ~10-50ms (depends on network)
+
+### Optimization Tips
+
+1. Use async transports for production
+2. Batch events when possible
+3. Disable stack traces for low-severity events
+4. Set appropriate metadata size limits
+5. Use connection pooling for transports
+
+## Testing Strategy
+
+### Unit Tests
+
+- Model validation
+- Decorator behavior
+- Service logic
+- Transport implementations
+
+### Integration Tests
+
+- End-to-end capture flow
+- Kafka integration
+- Configuration loading
+- Error scenarios
+
+### Mock Support
+
+All external dependencies can be mocked:
+
+- Kafka producer
+- File system
+- Database connections
+
+## Security Considerations
+
+### PII Protection
+
+- Avoid capturing sensitive data in metadata
+- Sanitize stack traces (optional)
+- Redact patterns (configurable)
+
+### Authentication
+
+- Kafka SASL support
+- Custom transport auth
+- Certificate-based auth
+
+## Future Enhancements
+
+1. **Metrics**: Prometheus/StatsD integration
+2. **Tracing**: OpenTelemetry support
+3. **Sampling**: Intelligent event sampling
+4. **Aggregation**: Event deduplication
+5. **Alerting**: Direct alerting integration
+6. **Dashboard**: Real-time event dashboard

peclibrary-0.7.0/MANIFEST.in

@@ -0,0 +1,8 @@
+include README.md
+include LICENSE
+include ARCHITECTURE.md
+include requirements.txt
+recursive-include src *.py
+recursive-include src py.typed
+recursive-exclude tests *
+recursive-exclude docs *