fastapi-observer 1.12.0__tar.gz → 1.14.0__tar.gz

{fastapi_observer-1.12.0/src/fastapi_observer.egg-info → fastapi_observer-1.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastapi-observer
-Version: 1.12.0
+Version: 1.14.0
 Summary: Zero-glue FastAPI observability with security presets and runtime controls
 Author-email: Vitaee <opensource@vitaee.dev>
 License-Expression: MIT
@@ -150,9 +150,9 @@ Explicit variables and Python arguments always win over profile defaults. Every
 - **Request IDs** taken from `X-Request-ID` when the caller is a trusted proxy, generated otherwise, returned in the response and passed on to the services you call, so one ID finds a request's logs in every service.
 - **Redaction on by default.** Passwords, tokens, API keys, `Authorization` and cookies are masked before anything is written. Presets cover stricter needs (`strict`, `pci`, `gdpr`).
 - **Prometheus metrics** labelled by route template (`/orders/{order_id}`), and 404s for paths no route matches share one `<unmatched>` label, so neither IDs nor scanners explode cardinality.
-- **OpenTelemetry traces** with `trace_id` and `span_id` in every log line, and optional log export over OTLP.
+- **OpenTelemetry traces**, from FastAPI's own support on 0.142 and later, with `trace_id` and `span_id` in every log line, and optional log export over OTLP, redacted like everything else.
 - **Non-blocking log writes.** Lines are written from a background thread, so a slow stdout never stalls the event loop.
-- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
+- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. Always on: a warning when the thread pool or a database connection pool is full, the usual cause when latency climbs while CPU stays low. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
 
 Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlchemy, `httpx` and `requests` tracing, Strawberry GraphQL and a Loguru bridge. See the [documentation](https://vitaee.github.io/FastapiObserver/).
 
@@ -162,14 +162,16 @@ Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlche
 
 ## How it compares
 
-| | fastapi-observer | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
-|---|---|---|---|---|
-| JSON log line per request | Built in | You write the middleware | No | A span per request |
-| Request ID on every log line | Built in | You wire it up | No | Trace IDs |
-| Redaction by default | Yes, with presets | You write a processor | No | Yes |
-| Prometheus `/metrics` | Built in | No | Through an exporter | No |
-| Traces | OpenTelemetry | No | Yes | Yes |
-| Where data goes | stdout, Prometheus and any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+| | fastapi-observer | FastAPI 0.142+ built in | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
+|---|---|---|---|---|---|
+| JSON log line per request | Built in | No, only errors as OTLP log records | You write the middleware | No | A span per request |
+| Request ID on every log line | Built in | No | You wire it up | No | Trace IDs |
+| Redaction by default | Yes, with presets | Signed-URL query parameters only | You write a processor | No | Yes |
+| Prometheus `/metrics` | Built in | Through an exporter | No | Through an exporter | No |
+| Traces | FastAPI's own on 0.142+, OpenTelemetry's instrumentation before | Yes, with spans for dependencies, the endpoint and serialization | No | Yes | Yes |
+| Where data goes | stdout, Prometheus and any OTLP backend | Any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+
+Since 0.142, FastAPI traces requests on its own once an OpenTelemetry provider is set up. fastapi-observer uses that for traces and adds the rest of the first column.
 
 **When not to use it:**
 

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/README.md

@@ -74,9 +74,9 @@ Explicit variables and Python arguments always win over profile defaults. Every
 - **Request IDs** taken from `X-Request-ID` when the caller is a trusted proxy, generated otherwise, returned in the response and passed on to the services you call, so one ID finds a request's logs in every service.
 - **Redaction on by default.** Passwords, tokens, API keys, `Authorization` and cookies are masked before anything is written. Presets cover stricter needs (`strict`, `pci`, `gdpr`).
 - **Prometheus metrics** labelled by route template (`/orders/{order_id}`), and 404s for paths no route matches share one `<unmatched>` label, so neither IDs nor scanners explode cardinality.
-- **OpenTelemetry traces** with `trace_id` and `span_id` in every log line, and optional log export over OTLP.
+- **OpenTelemetry traces**, from FastAPI's own support on 0.142 and later, with `trace_id` and `span_id` in every log line, and optional log export over OTLP, redacted like everything else.
 - **Non-blocking log writes.** Lines are written from a background thread, so a slow stdout never stalls the event loop.
-- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
+- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. Always on: a warning when the thread pool or a database connection pool is full, the usual cause when latency climbs while CPU stays low. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
 
 Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlchemy, `httpx` and `requests` tracing, Strawberry GraphQL and a Loguru bridge. See the [documentation](https://vitaee.github.io/FastapiObserver/).
 
@@ -86,14 +86,16 @@ Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlche
 
 ## How it compares
 
-| | fastapi-observer | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
-|---|---|---|---|---|
-| JSON log line per request | Built in | You write the middleware | No | A span per request |
-| Request ID on every log line | Built in | You wire it up | No | Trace IDs |
-| Redaction by default | Yes, with presets | You write a processor | No | Yes |
-| Prometheus `/metrics` | Built in | No | Through an exporter | No |
-| Traces | OpenTelemetry | No | Yes | Yes |
-| Where data goes | stdout, Prometheus and any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+| | fastapi-observer | FastAPI 0.142+ built in | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
+|---|---|---|---|---|---|
+| JSON log line per request | Built in | No, only errors as OTLP log records | You write the middleware | No | A span per request |
+| Request ID on every log line | Built in | No | You wire it up | No | Trace IDs |
+| Redaction by default | Yes, with presets | Signed-URL query parameters only | You write a processor | No | Yes |
+| Prometheus `/metrics` | Built in | Through an exporter | No | Through an exporter | No |
+| Traces | FastAPI's own on 0.142+, OpenTelemetry's instrumentation before | Yes, with spans for dependencies, the endpoint and serialization | No | Yes | Yes |
+| Where data goes | stdout, Prometheus and any OTLP backend | Any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+
+Since 0.142, FastAPI traces requests on its own once an OpenTelemetry provider is set up. fastapi-observer uses that for traces and adds the rest of the first column.
 
 **When not to use it:**
 

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0/src/fastapi_observer.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fastapi-observer
-Version: 1.12.0
+Version: 1.14.0
 Summary: Zero-glue FastAPI observability with security presets and runtime controls
 Author-email: Vitaee <opensource@vitaee.dev>
 License-Expression: MIT
@@ -150,9 +150,9 @@ Explicit variables and Python arguments always win over profile defaults. Every
 - **Request IDs** taken from `X-Request-ID` when the caller is a trusted proxy, generated otherwise, returned in the response and passed on to the services you call, so one ID finds a request's logs in every service.
 - **Redaction on by default.** Passwords, tokens, API keys, `Authorization` and cookies are masked before anything is written. Presets cover stricter needs (`strict`, `pci`, `gdpr`).
 - **Prometheus metrics** labelled by route template (`/orders/{order_id}`), and 404s for paths no route matches share one `<unmatched>` label, so neither IDs nor scanners explode cardinality.
-- **OpenTelemetry traces** with `trace_id` and `span_id` in every log line, and optional log export over OTLP.
+- **OpenTelemetry traces**, from FastAPI's own support on 0.142 and later, with `trace_id` and `span_id` in every log line, and optional log export over OTLP, redacted like everything else.
 - **Non-blocking log writes.** Lines are written from a background thread, so a slow stdout never stalls the event loop.
-- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
+- **Help finding what slows a request down.** Opt-in: a warning with the stack of any sync call that blocks the event loop, and `DEBUG` lines written only for requests that fail. Always on: a warning when the thread pool or a database connection pool is full, the usual cause when latency climbs while CPU stays low. With tracing on, each request line counts its queries and outbound HTTP calls, and a query run once per item (N+1) gets a warning. With metrics on, `/metrics` shows how many bytes of logs each route writes, and `LOG_SAMPLE_RATE` keeps a share of successful requests, chosen by trace ID, while failed and slow requests are always written in full.
 
 Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlchemy, `httpx` and `requests` tracing, Strawberry GraphQL and a Loguru bridge. See the [documentation](https://vitaee.github.io/FastapiObserver/).
 
@@ -162,14 +162,16 @@ Optional extras: runtime log-level control, a tamper-evident audit log, SQLAlche
 
 ## How it compares
 
-| | fastapi-observer | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
-|---|---|---|---|---|
-| JSON log line per request | Built in | You write the middleware | No | A span per request |
-| Request ID on every log line | Built in | You wire it up | No | Trace IDs |
-| Redaction by default | Yes, with presets | You write a processor | No | Yes |
-| Prometheus `/metrics` | Built in | No | Through an exporter | No |
-| Traces | OpenTelemetry | No | Yes | Yes |
-| Where data goes | stdout, Prometheus and any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+| | fastapi-observer | FastAPI 0.142+ built in | structlog or loguru | OpenTelemetry auto-instrumentation | Logfire |
+|---|---|---|---|---|---|
+| JSON log line per request | Built in | No, only errors as OTLP log records | You write the middleware | No | A span per request |
+| Request ID on every log line | Built in | No | You wire it up | No | Trace IDs |
+| Redaction by default | Yes, with presets | Signed-URL query parameters only | You write a processor | No | Yes |
+| Prometheus `/metrics` | Built in | Through an exporter | No | Through an exporter | No |
+| Traces | FastAPI's own on 0.142+, OpenTelemetry's instrumentation before | Yes, with spans for dependencies, the endpoint and serialization | No | Yes | Yes |
+| Where data goes | stdout, Prometheus and any OTLP backend | Any OTLP backend | stdout or files | Any OTLP backend | Logfire's platform |
+
+Since 0.142, FastAPI traces requests on its own once an OpenTelemetry provider is set up. fastapi-observer uses that for traces and adds the rest of the first column.
 
 **When not to use it:**
 

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapi_observer.egg-info/SOURCES.txt

@@ -15,6 +15,7 @@ src/fastapiobserver/fastapi.py
 src/fastapiobserver/loguru.py
 src/fastapiobserver/loop_watchdog.py
 src/fastapiobserver/plugins.py
+src/fastapiobserver/pools.py
 src/fastapiobserver/profiles.py
 src/fastapiobserver/propagation.py
 src/fastapiobserver/py.typed
@@ -98,6 +99,7 @@ tests/test_otel_logs_pipeline.py
 tests/test_otel_metrics_from_env.py
 tests/test_otlp_export_integration.py
 tests/test_plugin_isolation.py
+tests/test_pools.py
 tests/test_profiles.py
 tests/test_propagation.py
 tests/test_redaction_presets.py

fastapi_observer-1.14.0/src/fastapiobserver/_version.py

@@ -0,0 +1 @@
+__version__ = "1.14.0"

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/fastapi.py

@@ -35,6 +35,7 @@ from .otel import (
     install_otel_metrics,
 )
 from .otel.native import keep_native_logs_out
+from .pools import watch_db_pool, watch_thread_pool
 from .propagation import install_request_id_propagation
 from .security import SecurityPolicy, TrustedProxyPolicy
 
@@ -109,9 +110,13 @@ def install_observability(
     if otel_settings and otel_settings.enabled:
         install_otel(app, settings, otel_settings)
 
-    # --- 4b. Database tracing + SQLCommenter ---
-    if db_engine is not None and otel_settings and otel_settings.enabled:
-        _setup_database_tracing(db_engine, db_commenter_enabled, db_commenter_options)
+    # --- 4b. Database pools, tracing + SQLCommenter ---
+    if db_engine is not None:
+        engines = _as_engine_list(db_engine)
+        for engine in engines:
+            watch_db_pool(getattr(engine, "sync_engine", engine))
+        if otel_settings and otel_settings.enabled:
+            _setup_database_tracing(engines, db_commenter_enabled, db_commenter_options)
 
     # --- 5. OTel metrics ---
     if otel_metrics_settings and otel_metrics_settings.enabled:
@@ -143,6 +148,21 @@ def install_observability(
     if runtime_control_settings and runtime_control_settings.enabled:
         mount_control_plane(app, runtime_control_settings, trusted_proxy_policy)
 
+    if security_policy.redaction_mode == "hash" and security_policy.hash_key is None:
+        _LOGGER.warning(
+            "observability.redaction.hash_key_missing",
+            extra={
+                "event": {
+                    "message": (
+                        "Hashed values use a random key picked at startup, so they differ "
+                        "between processes and after a restart. Set OBS_REDACTION_HASH_KEY "
+                        "to keep them stable."
+                    ),
+                },
+                "_skip_enrichers": True,
+            },
+        )
+
     # --- 9. One line with the settings that took effect ---
     _LOGGER.info(
         "observability.configured",
@@ -262,8 +282,17 @@ def _setup_metrics_backend(
     return metrics_backend
 
 
+def _as_engine_list(db_engine: Any | Sequence[Any]) -> list[Any]:
+    if isinstance(db_engine, (str, bytes, bytearray)):
+        raise TypeError(
+            f"db_engine must be an SQLAlchemy Engine or sequence of Engines, "
+            f"not {type(db_engine).__name__}"
+        )
+    return list(db_engine) if isinstance(db_engine, Sequence) else [db_engine]
+
+
 def _setup_database_tracing(
-    db_engine: Any | Sequence[Any],
+    engines: list[Any],
     db_commenter_enabled: bool,
     db_commenter_options: dict[str, bool] | None,
 ) -> None:
@@ -272,13 +301,6 @@ def _setup_database_tracing(
         instrument_sqlalchemy_async,
     )
 
-    if isinstance(db_engine, (str, bytes, bytearray)):
-        raise TypeError(
-            f"db_engine must be an SQLAlchemy Engine or sequence of Engines, "
-            f"not {type(db_engine).__name__}"
-        )
-
-    engines = db_engine if isinstance(db_engine, Sequence) else [db_engine]
     for engine in engines:
         if hasattr(engine, "sync_engine"):
             instrument_sqlalchemy_async(
@@ -400,6 +422,7 @@ async def observability_lifespan(app: FastAPI) -> typing.AsyncGenerator[typing.A
     watchdog = None
     if state:
         _auto_discover_excluded_routes(app, state.settings)
+        watch_thread_pool()
         if state.settings.loop_block_threshold_ms:
             watchdog = EventLoopWatchdog(state.settings.loop_block_threshold_ms / 1000)
             watchdog.start()

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/logging/formatter.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 import hashlib
 import json
 import logging
-import re
+import traceback
 from datetime import datetime, timezone
+from types import TracebackType
 from typing import Any, Callable, Mapping
 
 from .._version import __version__
@@ -144,11 +145,11 @@ def _build_structured_error(
             "stacktrace": "",
         }
 
-    exc_type, exc_value, _exc_tb = exc_info
+    exc_type, exc_value, exc_tb = exc_info
     error_type = exc_type.__name__ if exc_type is not None else "Exception"
     error_message = str(exc_value) if exc_value is not None else ""
     stacktrace = formatter.formatException(exc_info)
-    fingerprint = _generate_error_fingerprint(error_type, stacktrace)
+    fingerprint = _generate_error_fingerprint(error_type, exc_tb)
 
     return {
         "type": error_type,
@@ -158,16 +159,13 @@ def _build_structured_error(
     }
 
 
-def _generate_error_fingerprint(error_type: str, stacktrace: str) -> str:
-    """Hash stack trace after stripping out environment-specific noise."""
-    if not stacktrace:
-        return hashlib.sha256(error_type.encode("utf-8")).hexdigest()[:32]
+def _generate_error_fingerprint(error_type: str, exc_tb: TracebackType | None) -> str:
+    """Hash the exception type and the file, function and source of each frame.
 
-    # Strip hexadecimal memory addresses
-    sanitized = re.sub(r"0x[0-9a-fA-F]+", "0x<ptr>", stacktrace)
-    # Strip exact line numbers to survive minor file refactoring
-    sanitized = re.sub(r"line \d+", "line <N>", sanitized)
-    
-    # Combine error type and sanitized trace to ensure distinct groupings
-    payload = f"{error_type}:{sanitized}"
-    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:32]
+    The message is left out, since it often holds IDs that differ each time the
+    same bug is hit, and so are line numbers, which shift with unrelated edits.
+    """
+    frames = "".join(
+        f"{frame.filename}:{frame.name}:{frame.line}\n" for frame in traceback.extract_tb(exc_tb)
+    )
+    return hashlib.sha256(f"{error_type}\n{frames}".encode("utf-8")).hexdigest()[:32]

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/logging/setup.py

@@ -4,6 +4,7 @@ import atexit
 import logging
 import logging.handlers
 import queue
+import threading
 from typing import Literal, cast
 
 from ..config import ObservabilitySettings
@@ -289,22 +290,16 @@ def _restart_queue_listener_in_child() -> None:
 
     listener = state._QUEUE_LISTENER
     if listener is not None:
-        # The inherited queue's Condition lock might be held by the parent
-        # or left in an inconsistent state across the fork. To reliably resume
-        # processing, we must create a fresh queue. We explicitly drain and 
-        # DROP inherited records because the parent process's listener thread 
-        # is already processing those same records in the parent process. 
-        # Re-queuing them here causes deterministic duplicate logs.
+        # Another thread of the parent may have held any lock at the moment of
+        # the fork, and no thread in the child will release it. So don't touch
+        # the inherited queue, not even to drain it: its records are the
+        # parent's to write. Give the child fresh locks and a fresh queue.
+        _LOG_QUEUE_TELEMETRY._lock = threading.Lock()
+        for breaker in state._SINK_CIRCUIT_BREAKERS:
+            breaker._lock = threading.Lock()
         old_queue = cast(queue.Queue[logging.LogRecord], listener.queue)
         new_queue: queue.Queue[logging.LogRecord] = queue.Queue(maxsize=old_queue.maxsize)
 
-        while not old_queue.empty():
-            try:
-                # Discard the record to prevent pre-fork replay duplication
-                _ = old_queue.get_nowait()
-            except queue.Empty:
-                break
-
         # Re-attach the new queue to the OverflowPolicyQueueHandler if present
         for handler in state._MANAGED_HANDLERS:
             from .queueing import OverflowPolicyQueueHandler

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/metrics/prometheus.py

@@ -186,17 +186,65 @@ class _LogQueueMetricsCollector:
             yield counter
 
 
-def _register_log_queue_metrics_collector() -> None:
-    # Per-worker queue stats would be meaningless once aggregated across processes.
+class _PoolMetricsCollector:
+    """Exposes thread pool and SQLAlchemy connection pool usage at scrape time."""
+
+    def collect(self) -> Iterator[Any]:
+        from prometheus_client.core import GaugeMetricFamily
+
+        from ..pools import db_pool_usage, thread_pool_usage
+
+        threads = thread_pool_usage()
+        if threads is not None:
+            busy, limit, waiting = threads
+            yield GaugeMetricFamily(
+                "fastapiobserver_threadpool_busy",
+                "Threads running sync endpoints and dependencies.",
+                value=busy,
+            )
+            yield GaugeMetricFamily(
+                "fastapiobserver_threadpool_limit",
+                "Threads AnyIO's default pool may run at once.",
+                value=limit,
+            )
+            yield GaugeMetricFamily(
+                "fastapiobserver_threadpool_waiting",
+                "Calls waiting for a thread.",
+                value=waiting,
+            )
+
+        db_pools = db_pool_usage()
+        if not db_pools:
+            return
+        checked_out = GaugeMetricFamily(
+            "fastapiobserver_db_pool_checked_out",
+            "Connections checked out of each SQLAlchemy pool.",
+            labels=["engine"],
+        )
+        limits = GaugeMetricFamily(
+            "fastapiobserver_db_pool_limit",
+            "Connections each SQLAlchemy pool may hand out: pool_size plus max_overflow.",
+            labels=["engine"],
+        )
+        for label, out, limit in db_pools:
+            checked_out.add_metric([label], out)
+            limits.add_metric([label], limit)
+        yield checked_out
+        yield limits
+
+
+def _register_collectors() -> None:
+    # Per-worker stats would be meaningless once aggregated across processes.
     if _multiprocess_dir():
         return
     from ..logging.volume import start_counting
 
     start_counting()
-    try:
-        _import_prometheus_client().REGISTRY.register(_LogQueueMetricsCollector())
-    except ValueError:
-        pass  # Already registered by an earlier backend build in this process.
+    for collector in (_LogQueueMetricsCollector(), _PoolMetricsCollector()):
+        try:
+            _import_prometheus_client().REGISTRY.register(collector)
+        except ValueError:
+            pass  # Already registered by an earlier backend build in this process.
 
 
 class PrometheusMetricsBackend:
@@ -288,7 +336,7 @@ def _build_prometheus_metrics_backend(
         environment=environment,
         exemplars_enabled=exemplars_enabled,
     )
-    _register_log_queue_metrics_collector()
+    _register_collectors()
     return backend
 
 

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/middleware/request_logging.py

@@ -4,11 +4,13 @@ import logging
 import re
 import time
 
+from anyio import get_cancelled_exc_class
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 
 from ..config import ObservabilitySettings
 from ..logging.held import keep_request_logs
 from ..metrics import MetricsBackend, NoopMetricsBackend
+from ..pools import check_pools
 from ..request_context import get_request_state
 from ..security import (
     SecurityPolicy,
@@ -62,6 +64,7 @@ class RequestLoggingMiddleware:
             await self.app(scope, receive, send)
             return
 
+        check_pools()
         start = time.perf_counter()
         raw_headers = scope.get("headers", [])
         client_ip, trusted_source = self.ip_resolver.resolve(scope, raw_headers)
@@ -84,6 +87,8 @@ class RequestLoggingMiddleware:
             max_length=self.security_policy.max_body_length,
         )
         had_error = False
+        cancelled = False
+        response_started = False
 
         async def receive_wrapper() -> Message:
             message = await receive()
@@ -91,8 +96,9 @@ class RequestLoggingMiddleware:
             return message
 
         async def send_wrapper(message: Message) -> None:
-            nonlocal status_code
+            nonlocal status_code, response_started
             if message["type"] == "http.response.start":
+                response_started = True
                 status_code = int(message.get("status", 500))
                 resp_headers = message.get("headers", [])
                 if self.security_policy.log_response_body:
@@ -122,6 +128,13 @@ class RequestLoggingMiddleware:
             captured_error = exc
             status_code = 500
             raise
+        except get_cancelled_exc_class():
+            # The client left, the server is shutting down or a timeout fired.
+            # Not a server error: without a response, use nginx's 499.
+            cancelled = True
+            if not response_started:
+                status_code = 499
+            raise
         finally:
             self._finalize_request(
                 scope=scope,
@@ -134,6 +147,7 @@ class RequestLoggingMiddleware:
                 response_body_capture=response_body_capture,
                 captured_error=captured_error,
                 had_error=had_error,
+                cancelled=cancelled,
             )
 
     def _finalize_request(
@@ -148,10 +162,13 @@ class RequestLoggingMiddleware:
         response_body_capture: _BodyCapture,
         captured_error: Exception | None,
         had_error: bool,
+        cancelled: bool,
     ) -> None:
         duration_seconds = max(0.0, time.perf_counter() - start)
         if had_error:
             level, message = logging.ERROR, "request.failed"
+        elif cancelled:
+            level, message = logging.WARNING, "request.cancelled"
         elif status_code >= 500:
             level, message = logging.ERROR, "request.server_error"
         elif status_code >= 400:

fastapi_observer-1.14.0/src/fastapiobserver/pools.py

@@ -0,0 +1,99 @@
+"""Full pools: requests wait, latency climbs, CPU stays low.
+
+Two pools make requests wait without using CPU: AnyIO's thread pool, which runs
+sync endpoints and dependencies, and SQLAlchemy connection pools. Neither AnyIO
+nor SQLAlchemy has a hook for how long one call waited, so this reports when a
+pool is full: a warning at most once a minute per pool, checked as each request
+starts, and gauges on ``/metrics``. Requests still start when every thread and
+connection is stuck, so a deadlock is reported too.
+"""
+
+from __future__ import annotations
+
+import logging
+import math
+import time
+import weakref
+from typing import Any
+
+_LOGGER = logging.getLogger("fastapiobserver.pools")
+# ponytail: fixed; make it a setting if once a minute is too noisy or too quiet.
+_WARN_INTERVAL_SECONDS = 60.0
+
+_thread_limiter: Any = None  # AnyIO's default limiter, captured on the event loop
+_db_pools: weakref.WeakKeyDictionary[Any, str] = weakref.WeakKeyDictionary()  # engine: label
+_last_warned: dict[str, float] = {}
+
+
+def watch_thread_pool() -> None:
+    """Remember AnyIO's default thread limiter. Call it on the event loop."""
+    global _thread_limiter
+    from anyio.to_thread import current_default_thread_limiter
+
+    _thread_limiter = current_default_thread_limiter()
+
+
+def watch_db_pool(engine: Any) -> None:
+    """Watch a sync SQLAlchemy engine's pool, if it has a limit (``QueuePool``)."""
+    if callable(getattr(engine.pool, "checkedout", None)):
+        _db_pools[engine] = engine.url.render_as_string(hide_password=True)
+
+
+def thread_pool_usage() -> tuple[int, float, int] | None:
+    """Busy threads, the limit and calls waiting; None until the app has started."""
+    if _thread_limiter is None:
+        return None
+    stats = _thread_limiter.statistics()
+    return stats.borrowed_tokens, stats.total_tokens, stats.tasks_waiting
+
+
+def db_pool_usage() -> list[tuple[str, int, float]]:
+    """Each watched engine's label, checked-out connections and limit."""
+    return [
+        (label, engine.pool.checkedout(), _db_pool_limit(engine.pool))
+        for engine, label in list(_db_pools.items())
+    ]
+
+
+def check_pools() -> None:
+    """Warn about full pools. Called as each request starts."""
+    limiter = _thread_limiter
+    # Cheap test first; statistics() copies the borrower list.
+    if limiter is not None and limiter.borrowed_tokens >= limiter.total_tokens:
+        busy, limit, waiting = thread_pool_usage() or (0, 0, 0)
+        if waiting:
+            _warn(
+                "threadpool.saturated",
+                "threadpool",
+                {"busy": busy, "limit": limit, "waiting": waiting},
+            )
+    for label, checked_out, limit in db_pool_usage():
+        if checked_out >= limit:
+            _warn(
+                "db.pool_exhausted",
+                label,
+                {"engine": label, "checked_out": checked_out, "limit": limit},
+            )
+
+
+def _db_pool_limit(pool: Any) -> float:
+    # max_overflow has no public accessor; without it, or at -1, there's no limit.
+    max_overflow = getattr(pool, "_max_overflow", -1)
+    return math.inf if max_overflow < 0 else pool.size() + max_overflow
+
+
+def _warn(message: str, pool: str, event: dict[str, Any]) -> None:
+    now = time.monotonic()
+    if now - _last_warned.get(pool, -math.inf) < _WARN_INTERVAL_SECONDS:
+        return
+    _last_warned[pool] = now
+    _LOGGER.warning(message, extra={"event": event, "_skip_enrichers": True})
+
+
+__all__ = [
+    "check_pools",
+    "db_pool_usage",
+    "thread_pool_usage",
+    "watch_db_pool",
+    "watch_thread_pool",
+]

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/security/policy.py

@@ -1,7 +1,7 @@
 import os
 from typing import Any, Literal
 
-from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator
+from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationInfo, field_validator
 from pydantic_settings import BaseSettings
 
 from ..utils import EnvLoadable, InternalSettingsBase, parse_csv
@@ -88,6 +88,7 @@ _SECURITY_POLICY_ENV_MAP = {
     "redacted_headers": "OBS_REDACTED_HEADERS",
     "redaction_mode": "OBS_REDACTION_MODE",
     "mask_text": "OBS_MASK_TEXT",
+    "hash_key": "OBS_REDACTION_HASH_KEY",
     "log_request_body": "OBS_LOG_REQUEST_BODY",
     "log_response_body": "OBS_LOG_RESPONSE_BODY",
     "max_body_length": "OBS_MAX_BODY_LENGTH",
@@ -112,6 +113,9 @@ class SecurityPolicy(EnvLoadable, BaseModel):
     redacted_headers: tuple[str, ...] = DEFAULT_REDACTED_HEADERS
     redaction_mode: RedactionMode = "mask"
     mask_text: str = "***"
+    # HMAC key for the hash mode. Without one, a random key is picked at import,
+    # so the same value hashes differently in other processes and after a restart.
+    hash_key: SecretStr | None = None
     log_request_body: bool = False
     log_response_body: bool = False
     max_body_length: int = Field(default=256, gt=0)
@@ -233,6 +237,7 @@ class _SecurityPolicySettings(InternalSettingsBase):
         validation_alias="OBS_REDACTION_MODE",
     )
     mask_text: str | None = Field(default=None, validation_alias="OBS_MASK_TEXT")
+    hash_key: SecretStr | None = Field(default=None, validation_alias="OBS_REDACTION_HASH_KEY")
     log_request_body: bool | None = Field(
         default=None,
         validation_alias="OBS_LOG_REQUEST_BODY",

{fastapi_observer-1.12.0 → fastapi_observer-1.14.0}/src/fastapiobserver/security/redaction.py

@@ -1,5 +1,7 @@
 import functools
 import hashlib
+import hmac
+import secrets
 from collections.abc import Mapping
 from typing import Any
 
@@ -8,6 +10,7 @@ from .policy import _DROP, SecurityPolicy
 
 MAX_SANITIZE_DEPTH = 32
 _SCALAR_TYPES = frozenset({str, int, float, bool})
+_PROCESS_KEY = secrets.token_bytes(32)  # for policies without a hash_key
 
 
 @functools.lru_cache(maxsize=64)
@@ -33,8 +36,11 @@ def _redact_value(value: Any, policy: SecurityPolicy) -> Any:
     if policy.redaction_mode == "drop":
         return _DROP
     if policy.redaction_mode == "hash":
-        digest = hashlib.sha256(str(value).encode("utf-8")).hexdigest()
-        return f"sha256:{digest}"
+        # Keyed: a plain hash of an email or phone number is reversed by
+        # hashing candidates until one matches.
+        key = policy.hash_key.get_secret_value().encode() if policy.hash_key else _PROCESS_KEY
+        digest = hmac.new(key, str(value).encode("utf-8"), hashlib.sha256).hexdigest()
+        return f"hmac-sha256:{digest}"
     return policy.mask_text