aice-observability 0.1.0__py3-none-any.whl

aice_observability-0.1.0.dist-info/METADATA

@@ -0,0 +1,746 @@
+Metadata-Version: 2.4
+Name: aice-observability
+Version: 0.1.0
+Summary: Unified OpenTelemetry-based observability layer for internal Python services
+Author-email: Arnold Opiyo <arnoldopiyo@adanianlabs.io>
+Keywords: metrics,observability,opentelemetry,telemetry,tracing
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.11
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27.0
+Requires-Dist: opentelemetry-instrumentation-django>=0.64b0
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.48b0
+Requires-Dist: opentelemetry-instrumentation-flask>=0.64b0
+Requires-Dist: opentelemetry-instrumentation-httpx>=0.48b0
+Requires-Dist: opentelemetry-instrumentation-logging>=0.48b0
+Requires-Dist: opentelemetry-instrumentation-psycopg2>=0.64b0
+Requires-Dist: opentelemetry-instrumentation-redis>=0.48b0
+Requires-Dist: opentelemetry-instrumentation-sqlalchemy>=0.48b0
+Requires-Dist: opentelemetry-instrumentation-system-metrics>=0.48b0
+Requires-Dist: opentelemetry-sdk>=1.27.0
+Provides-Extra: dev
+Requires-Dist: fastapi>=0.100.0; extra == 'dev'
+Requires-Dist: httpx>=0.24.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.3.0; extra == 'dev'
+Requires-Dist: redis>=5.0.0; extra == 'dev'
+Requires-Dist: sqlalchemy>=2.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+Unified OpenTelemetry bootstrap for internal Python services. The objective is to provide every service with a consistent operational view without requiring teams to build dashboards from scratch.
+
+## Usage
+
+```python
+from fastapi import FastAPI
+from observability import Observability
+
+app = FastAPI()
+
+obs = (
+    Observability()
+    .service("data-quality-service", version="2.0.0")
+    .start()                        # must come before instrument_*() calls
+    .instrument_fastapi(app)
+    .instrument_sqlalchemy(engine)
+)
+```
+
+`.start()` must be called before any `instrument_*()` method. OpenTelemetry's
+instrumentation libraries bind to whatever provider is globally registered
+at the moment they run; calling them before `.start()` would silently do
+nothing.
+
+httpx is instrumented automatically inside `.start()`, since
+they need no app-specific object.
+
+## Configuration (Environment Variables)
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes, unless all telemetry is disabled | none | OTLP HTTP endpoint used to export traces, metrics, and logs. Examples: `http://192.168.100.254:4318` (Collector), `https://otlp-gateway-prod-us-central-0.grafana.net/otlp` (Grafana Cloud). |
+| `OTEL_EXPORTER_OTLP_HEADERS` | No | none | Optional authentication headers in `key=value,key=value` format. Required by many hosted OTLP providers. |
+| `OTEL_SERVICE_NAME` | Yes | none | Logical service name reported to the observability backend (for example `dq-service`). |
+| `OTEL_ENVIRONMENT` | No | `development` | Deployment environment such as `development`, `staging`, or `production`. |
+| `OTEL_TRACES_ENABLED` | No | `true` | Enable or disable distributed tracing. |
+| `OTEL_METRICS_ENABLED` | No | `true` | Enable or disable metrics collection. |
+| `OTEL_LOGS_ENABLED` | No | `true` | Enable or disable log export. |
+| `OTEL_LOG_LEVEL` | No | `INFO` | Minimum application log level captured by the logging pipeline. |
+
+### Example: Self-hosted OpenTelemetry Collector
+
+```env
+OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.100.254:4318
+OTEL_SERVICE_NAME=dq-service
+OTEL_ENVIRONMENT=development
+OTEL_TRACES_ENABLED=true
+OTEL_METRICS_ENABLED=true
+OTEL_LOGS_ENABLED=true
+```
+
+
+# Standard Observability Dashboards
+
+
+The dashboards are designed around operational questions rather than telemetry types. Instead of exposing every metric collected by OpenTelemetry, each dashboard focuses on answering a specific set of questions that engineers, tech leads, and platform teams routinely ask.
+
+Version 1 of the observability platform includes the following dashboards:
+
+1. Service Overview
+2. Infrastructure
+3. HTTP/API
+4. Database (conditional)
+5. External Services (HTTP Client)
+6. Logs
+7. Traces
+
+---
+
+# 1. Service Overview Dashboard
+
+## Purpose
+
+The Service Overview dashboard is the primary landing page for every service.
+
+It provides a high level summary of service health, traffic, latency, failures, and operational status.
+
+This dashboard should answer the following questions within a few seconds:
+
+* Is the service healthy?
+* Is it currently receiving traffic?
+* Has latency increased?
+* Are users experiencing failures?
+* Should I investigate further?
+
+## Target Audience
+
+* Tech Leads
+* Backend Engineers
+* DevOps Engineers
+* SREs
+* Engineering Managers
+
+## Panels
+
+### Availability
+
+Displays service uptime and overall availability.
+
+Purpose:
+
+Quickly determine whether the service is operational.
+
+---
+
+### Request Rate (Requests Per Second)
+
+Displays incoming request volume.
+
+Purpose:
+
+Identify traffic spikes, traffic drops, and overall load.
+
+---
+
+### Active Requests
+
+Shows the number of requests currently being processed.
+
+Purpose:
+
+Detect request buildup and possible bottlenecks.
+
+---
+
+### Error Rate
+
+Percentage and total number of failed requests.
+
+Purpose:
+
+Quickly identify service degradation.
+
+---
+
+### Average Response Time
+
+Average latency across all requests.
+
+Purpose:
+
+Monitor overall responsiveness.
+
+---
+
+### P95 Latency
+
+95% of requests complete faster than this value.
+
+Purpose:
+
+Represents the experience of almost all users.
+
+---
+
+### P99 Latency
+
+99% of requests complete faster than this value.
+
+Purpose:
+
+Highlights long-tail latency issues.
+
+---
+
+### HTTP Status Code Distribution
+
+Breakdown of
+
+* 2xx
+* 3xx
+* 4xx
+* 5xx
+
+Purpose:
+
+Identify client errors versus server failures.
+
+---
+
+### Top Failing Endpoints
+
+Endpoints with the highest error count.
+
+Purpose:
+
+Quickly identify problematic APIs.
+
+---
+
+### Slowest Endpoints
+
+Endpoints ranked by average latency.
+
+Purpose:
+
+Locate performance bottlenecks.
+
+---
+
+### Service Metadata
+
+Displays
+
+* Service Name
+* Version
+* Environment
+* Instance Count
+* Last Restart Time
+
+Purpose:
+
+Provide deployment context during incident investigations.
+
+---
+
+### Quick Navigation
+
+Links directly to
+
+* Logs Dashboard
+* Traces Dashboard
+* Infrastructure Dashboard
+
+Purpose:
+
+Reduce investigation time.
+
+---
+
+# 2. Infrastructure Dashboard
+
+## Purpose
+
+The Infrastructure dashboard monitors resource utilization of the application process and host.
+
+It helps determine whether performance issues originate from application logic or infrastructure limitations.
+
+## Target Audience
+
+* Platform Engineers
+* DevOps Engineers
+* SREs
+
+## Panels
+
+### CPU Usage
+
+CPU utilization over time.
+
+---
+
+### Memory Usage
+
+Current memory consumption.
+
+---
+
+### Memory Utilization
+
+Memory usage as a percentage of available memory.
+
+---
+
+### Disk Usage
+
+Disk space utilization.
+
+---
+
+### Disk I/O
+
+Read and write throughput.
+
+---
+
+### Network Throughput
+
+Incoming and outgoing traffic.
+
+---
+
+### File Descriptors
+
+Number of open file descriptors.
+
+---
+
+### Thread Count
+
+Active application threads.
+
+---
+
+### Garbage Collection Activity
+
+Displays
+
+* Collection frequency
+* Objects collected
+* GC pause duration
+
+Purpose:
+
+Identify memory pressure.
+
+---
+
+### Process Restarts
+
+Detect unexpected application restarts.
+
+---
+
+# 3. HTTP/API Dashboard
+
+## Purpose
+
+Provides endpoint-level visibility into API behaviour.
+
+Rather than viewing the service as a whole, this dashboard focuses on individual routes.
+
+## Target Audience
+
+* Backend Engineers
+* API Owners
+
+## Panels
+
+### Requests Per Second by Endpoint
+
+---
+
+### Average Latency by Endpoint
+
+---
+
+### P95 Latency by Endpoint
+
+---
+
+### P99 Latency by Endpoint
+
+---
+
+### Error Rate by Endpoint
+
+---
+
+### Slowest Endpoints
+
+---
+
+### Largest Responses
+
+---
+
+### Largest Requests
+
+---
+
+### Request Duration Heatmap
+
+Visualizes latency distribution across all requests.
+
+---
+
+### Endpoint Ranking
+
+Ranks endpoints by
+
+* Traffic
+* Errors
+* Latency
+
+---
+
+# 4. Database Dashboard
+
+## Purpose
+
+Provides visibility into database interactions.
+
+This dashboard is displayed only when database instrumentation is enabled.
+
+Supported instrumentations include
+
+* SQLAlchemy
+* psycopg2
+
+## Target Audience
+
+* Backend Engineers
+* Database Administrators
+
+## Panels
+
+### Queries Per Second
+
+---
+
+### Average Query Duration
+
+---
+
+### Slowest Queries
+
+---
+
+### Database Error Rate
+
+---
+
+### Connection Pool Utilization
+
+---
+
+### Connection Pool Wait Time
+
+---
+
+### Pool Saturation
+
+---
+
+### Failed Queries
+
+---
+
+### Transaction Duration
+
+---
+
+### Database Retries
+
+---
+
+## Conditional Display
+
+Visible only if one of the following instrumentations is enabled.
+
+* SQLAlchemy
+* psycopg2
+
+---
+
+# 5. External Services Dashboard
+
+## Purpose
+
+Provides visibility into outbound HTTP requests.
+
+This dashboard helps identify slow or failing third-party services.
+
+The dashboard is powered by HTTPX instrumentation.
+
+## Target Audience
+
+* Backend Engineers
+* Tech Leads
+
+## Panels
+
+### Outbound Requests Per Second
+
+---
+
+### Average Outbound Latency
+
+---
+
+### P95 Outbound Latency
+
+---
+
+### Error Rate by Remote Service
+
+---
+
+### Timeout Count
+
+---
+
+### Retry Count
+
+---
+
+### Top Slowest Remote Hosts
+
+---
+
+### Request Distribution by Host
+
+---
+
+### External Dependency Health
+
+Summarizes
+
+* Success Rate
+* Failure Rate
+* Average Latency
+
+---
+
+# 6. Logs Dashboard
+
+## Purpose
+
+Provides centralized log exploration through Grafana and Loki.
+
+The dashboard is intended for investigation after metrics indicate abnormal behaviour.
+
+## Target Audience
+
+* All Engineers
+
+## Panels
+
+### Live Logs
+
+---
+
+### Error Logs
+
+---
+
+### Warning Logs
+
+---
+
+### Log Volume Over Time
+
+---
+
+### Log Levels
+
+Distribution of
+
+* DEBUG
+* INFO
+* WARNING
+* ERROR
+* CRITICAL
+
+---
+
+### Service Filter
+
+---
+
+### Search
+
+Supports free text search.
+
+---
+
+### Correlation Links
+
+Jump directly to related traces.
+
+---
+
+# 7. Traces Dashboard
+
+## Purpose
+
+Provides distributed tracing through Jaeger.
+
+The dashboard allows engineers to follow requests across services.
+
+## Target Audience
+
+* Backend Engineers
+* Platform Engineers
+* SREs
+
+## Panels
+
+### Recent Traces
+
+---
+
+### Slowest Traces
+
+---
+
+### Error Traces
+
+---
+
+### Trace Duration Distribution
+
+---
+
+### Service Dependency Graph
+
+---
+
+### Span Timeline
+
+---
+
+### Failed Spans
+
+---
+
+### Trace Search
+
+Supports filtering by
+
+* Service
+* Endpoint
+* Duration
+* Status
+
+---
+
+### Correlated Logs
+
+Jump directly to matching log entries.
+
+---
+
+# Dashboard Design Principles
+
+Every dashboard should follow these principles.
+
+## Minimal by Default
+
+Dashboards should expose only the information required for operational decisions.
+
+Avoid displaying every metric collected by OpenTelemetry.
+
+---
+
+## Progressive Investigation
+
+Dashboards should naturally lead the user from high level health to detailed diagnostics.
+
+Typical investigation flow:
+
+Service Overview
+
+↓
+
+HTTP/API
+
+↓
+
+Database or External Services
+
+↓
+
+Logs
+
+↓
+
+Traces
+
+---
+
+## Consistency
+
+All services should expose the same dashboard structure.
+
+This reduces cognitive load when switching between projects.
+
+---
+
+## Conditional Sections
+
+Dashboards should only display panels relevant to the service.
+
+Examples:
+
+A service without Redis should not display Redis metrics.
+
+A service without SQLAlchemy should not display SQL metrics.
+
+A service without HTTP clients should not display outbound request metrics.
+
+---
+
+## Standard Labels
+
+All dashboards should support filtering using common resource attributes.
+
+Examples include:
+
+* Service Name
+* Environment
+* Version
+* Instance
+* Endpoint
+* HTTP Method
+* Database System
+* Remote Host
+
+---
+
+# Future Dashboard Roadmap
+
+The following dashboards are planned for future releases.
+
+* Redis Dashboard
+* Background Jobs Dashboard
+* AI/LLM Dashboard
+* Cache Dashboard
+* Storage Dashboard
+* Security Dashboard
+* Business Metrics Dashboard
+
+