ganicas-package 0.1.3__py3-none-any.whl → 0.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,443 @@
1
+ Metadata-Version: 2.1
2
+ Name: ganicas-package
3
+ Version: 0.2.0
4
+ Summary: Ganicas internal Python package for structured logging and utilities.
5
+ Keywords: logging,utilities,internal-package,structlog,middleware,fastapi,flask
6
+ Author: Ganicas
7
+ Requires-Python: >=3.11,<4.0
8
+ Classifier: Development Status :: 3 - Alpha
9
+ Classifier: Framework :: FastAPI
10
+ Classifier: Framework :: Flask
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: Other/Proprietary License
13
+ Classifier: Natural Language :: English
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Topic :: System :: Logging
18
+ Requires-Dist: fastapi (>=0.114.2,<0.116.0)
19
+ Requires-Dist: flask (>=2.2.0,<3.0.0)
20
+ Requires-Dist: httpx (>=0.28.1,<0.29.0)
21
+ Requires-Dist: python-json-logger (>=3.2.1,<4.0.0)
22
+ Requires-Dist: structlog (>=24.4.0,<25.0.0)
23
+ Description-Content-Type: text/markdown
24
+
25
+ # Ganicas Utils
26
+
27
+ [![Python Version](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
28
+ [![License](https://img.shields.io/badge/license-Proprietary-red.svg)](LICENSE)
29
+ [![Code Coverage](https://img.shields.io/badge/coverage-99%25-brightgreen.svg)](https://github.com/ganicas/ganicas_utils)
30
+
31
+ **Ganicas Utils** is an internal Python package providing structured logging utilities and middleware for Flask and FastAPI applications. Built on top of [structlog](https://www.structlog.org/), it enables production-ready, context-aware logging with minimal configuration.
32
+
33
+ ---
34
+
35
+ ## 📋 Table of Contents
36
+
37
+ - [Features](#-features)
38
+ - [Installation](#-installation)
39
+ - [Quick Start](#-quick-start)
40
+ - [Structured Logging](#-structured-logging)
41
+ - [Basic Configuration](#basic-configuration)
42
+ - [Production Configuration](#production-configuration)
43
+ - [Middleware](#-middleware)
44
+ - [Flask Middleware](#flask-middleware)
45
+ - [FastAPI Middleware](#fastapi-middleware)
46
+ - [Request Logging Middleware](#request-logging-middleware)
47
+ - [Why Structured Logging?](#-why-structured-logging)
48
+ - [Development](#-development)
49
+ - [License](#-license)
50
+
51
+ ---
52
+
53
+ ## ✨ Features
54
+
55
+ - **Structured Logging**: JSON-formatted logs for easy parsing by log aggregation tools (ELK, Datadog, Grafana Loki)
56
+ - **Context Management**: Automatic request context binding (request_id, IP, user_agent, etc.)
57
+ - **Flask & FastAPI Support**: Ready-to-use middleware for both frameworks
58
+ - **Advanced Request Logging**: Comprehensive ASGI middleware with:
59
+ - Automatic request/response logging
60
+ - Sensitive header sanitization
61
+ - Slow request detection
62
+ - Sampling for high-traffic endpoints
63
+ - Exception tracking with full context
64
+ - Distributed tracing support (traceparent header)
65
+ - **Production Ready**: Battle-tested with 99% code coverage
66
+
67
+ ---
68
+
69
+ ## 📦 Installation
70
+
71
+ ```bash
72
+ pip install ganicas-package
73
+ ```
74
+
75
+ Or with Poetry:
76
+
77
+ ```bash
78
+ poetry add ganicas-package
79
+ ```
80
+
81
+ ---
82
+
83
+ ## 🚀 Quick Start
84
+
85
+ ### Basic Configuration
86
+
87
+ Replace `logger = logging.getLogger(__name__)` with `logger = structlog.get_logger(__name__)`:
88
+
89
+ ```python
90
+ from ganicas_utils.logging import LoggingConfigurator
91
+ from ganicas_utils.config import Config
92
+ import structlog
93
+
94
+ config = Config()
95
+
96
+ LoggingConfigurator(
97
+ service_name=config.APP_NAME,
98
+ log_level='INFO',
99
+ setup_logging_dict=True
100
+ ).configure_structlog(
101
+ formatter='plain_console',
102
+ formatter_std_lib='plain_console'
103
+ )
104
+
105
+ logger = structlog.get_logger(__name__)
106
+ logger.info("Application started", version="1.0.0", environment="production")
107
+ ```
108
+
109
+ ![basic example](images/plain_console_logger.png)
110
+
111
+ ---
112
+
113
+ ## 📊 Structured Logging
114
+
115
+ ### Production Configuration
116
+
117
+ For production environments, use JSON formatting for machine-readable logs:
118
+
119
+ ```python
120
+ from ganicas_utils.logging import LoggingConfigurator
121
+ from ganicas_utils.config import Config
122
+ import structlog
123
+
124
+ config = Config()
125
+
126
+ LoggingConfigurator(
127
+ service_name=config.APP_NAME,
128
+ log_level='INFO',
129
+ setup_logging_dict=True
130
+ ).configure_structlog(
131
+ formatter='json_formatter',
132
+ formatter_std_lib='json_formatter'
133
+ )
134
+
135
+ logger = structlog.get_logger(__name__)
136
+ logger.info("User login", user_id=12345, ip_address="192.168.1.1")
137
+ logger.warning("High memory usage", memory_percent=85.5, threshold=80)
138
+ logger.error("Database connection failed", db_host="localhost", error_code="CONN_REFUSED")
139
+
140
+ try:
141
+ result = 1 / 0
142
+ except ZeroDivisionError:
143
+ logger.exception("Division by zero error", operation="calculate_ratio")
144
+ ```
145
+
146
+ ![logger with different keys](images/json_logger.png)
147
+
148
+
149
+ ---
150
+
151
+ ## 🔧 Middleware
152
+
153
+ ### Flask Middleware
154
+
155
+ The `FlaskRequestContextMiddleware` automatically adds request context to all logs:
156
+
157
+ ```python
158
+ import uuid
159
+ from flask import Flask
160
+ from ganicas_utils.logging import LoggingConfigurator
161
+ from ganicas_utils.logging.middlewares import FlaskRequestContextMiddleware
162
+ from ganicas_utils.config import Config
163
+ import structlog
164
+
165
+ config = Config()
166
+
167
+ LoggingConfigurator(
168
+ service_name=config.APP_NAME,
169
+ log_level="INFO",
170
+ setup_logging_dict=True,
171
+ ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
172
+
173
+ logger = structlog.get_logger(__name__)
174
+
175
+ app = Flask(__name__)
176
+ app.wsgi_app = FlaskRequestContextMiddleware(app.wsgi_app)
177
+
178
+ @app.route("/")
179
+ def home():
180
+ logger.info("Processing request") # Automatically includes request_id, method, path
181
+ return "Hello, World!"
182
+
183
+ if __name__ == "__main__":
184
+ app.run()
185
+ ```
186
+
187
+ ![logger with context flask](images/flask_logger_with_context.png)
188
+
189
+ **Automatic context injection:**
190
+ - `request_id` - Unique identifier for each request
191
+ - `request_method` - HTTP method (GET, POST, etc.)
192
+ - `request_path` - Request URL path
193
+
194
+ ---
195
+
196
+ ### FastAPI Middleware
197
+
198
+ #### Basic Context Middleware
199
+
200
+ For simple request context binding, use `FastAPIRequestContextMiddleware`:
201
+
202
+ ```python
203
+ from fastapi import FastAPI
204
+ from ganicas_utils.logging import LoggingConfigurator
205
+ from ganicas_utils.logging.middlewares import FastAPIRequestContextMiddleware
206
+ from ganicas_utils.config import Config
207
+ import structlog
208
+
209
+ config = Config()
210
+
211
+ LoggingConfigurator(
212
+ service_name=config.APP_NAME,
213
+ log_level="INFO",
214
+ setup_logging_dict=True,
215
+ ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
216
+
217
+ logger = structlog.get_logger(__name__)
218
+ app = FastAPI()
219
+ app.add_middleware(FastAPIRequestContextMiddleware)
220
+
221
+ @app.get("/")
222
+ async def root():
223
+ logger.info("Processing request") # Automatically includes request context
224
+ return {"message": "Hello World"}
225
+ ```
226
+
227
+ ![logger with context fastapi](images/fastapi_logger_with_context.png)
228
+
229
+ ---
230
+
231
+ ### Request Logging Middleware
232
+
233
+ For production-grade request/response logging with advanced features, use `RequestLoggingMiddleware`:
234
+
235
+ ```python
236
+ from fastapi import FastAPI
237
+ from ganicas_utils.logging import LoggingConfigurator
238
+ from ganicas_utils.logging.middlewares import RequestLoggingMiddleware
239
+ import structlog
240
+
241
+ LoggingConfigurator(
242
+ service_name="my-api",
243
+ log_level="INFO",
244
+ setup_logging_dict=True,
245
+ ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
246
+
247
+ app = FastAPI()
248
+
249
+ # Add comprehensive request logging
250
+ app.add_middleware(
251
+ RequestLoggingMiddleware,
252
+ slow_request_threshold_ms=1000, # Warn on requests > 1s
253
+ propagate_request_id=True, # Add request_id to response headers
254
+ skip_paths={"/healthz", "/metrics"}, # Don't log health checks
255
+ sample_2xx_rate=0.1, # Sample 10% of successful requests
256
+ )
257
+
258
+ @app.get("/api/users/{user_id}")
259
+ async def get_user(user_id: int):
260
+ return {"user_id": user_id, "name": "John Doe"}
261
+ ```
262
+
263
+ #### Features
264
+
265
+ **Automatic Logging:**
266
+ - `request.start` - Logs when request begins
267
+ - `request.end` - Logs when request completes (with status, duration, size)
268
+ - `request.exception` - Logs unhandled exceptions with full traceback
269
+
270
+ **Logged Information:**
271
+ - Request: method, path, query params, client IP, user agent, content type/length
272
+ - Response: status code, size, content type, duration
273
+ - Headers: Sanitized request/response headers (for 4xx/5xx errors)
274
+ - Performance: Request duration, slow request detection
275
+
276
+ **Security:**
277
+ - Automatic sanitization of sensitive headers (`Authorization`, `Cookie`, `X-API-Key`)
278
+ - Authorization header preserves scheme: `Bearer ***` instead of exposing tokens
279
+ - No request/response body logging (only sizes)
280
+
281
+ **Performance Optimization:**
282
+ - Skip logging for health checks and metrics endpoints
283
+ - Sample successful requests to reduce log volume
284
+ - Skip OPTIONS requests
285
+ - Configurable path prefixes to skip
286
+
287
+ **Distributed Tracing:**
288
+ - Supports W3C `traceparent` header
289
+ - Falls back to `x-request-id` or `x-amzn-trace-id`
290
+ - Propagates request_id to response headers
291
+
292
+ #### Configuration Options
293
+
294
+ | Parameter | Type | Default | Description |
295
+ |-----------|------|---------|-------------|
296
+ | `logger` | `structlog.BoundLoggerBase` | `structlog.get_logger("http")` | Custom logger instance |
297
+ | `slow_request_threshold_ms` | `int` | `None` | Threshold in ms to flag slow requests |
298
+ | `propagate_request_id` | `bool` | `True` | Add `x-request-id` to response headers |
299
+ | `skip_paths` | `set[str]` | `{"/healthz", "/metrics"}` | Exact paths to skip logging |
300
+ | `skip_prefixes` | `tuple[str, ...]` | `("/metrics",)` | Path prefixes to skip logging |
301
+ | `sample_2xx_rate` | `float` | `None` | Sample rate for 2xx/3xx responses (0.0-1.0) |
302
+
303
+ #### Example Logs
304
+
305
+ **Successful Request:**
306
+ ```json
307
+ {
308
+ "event": "request.end",
309
+ "request_id": "550e8400-e29b-41d4-a716-446655440000",
310
+ "method": "GET",
311
+ "path": "/api/users/123",
312
+ "status_code": 200,
313
+ "duration_ms": 45,
314
+ "response_size": 256,
315
+ "client_ip": "192.168.1.100",
316
+ "user_agent": "Mozilla/5.0...",
317
+ "level": "info"
318
+ }
319
+ ```
320
+
321
+ **Slow Request Warning:**
322
+ ```json
323
+ {
324
+ "event": "request.end",
325
+ "request_id": "550e8400-e29b-41d4-a716-446655440001",
326
+ "method": "POST",
327
+ "path": "/api/process",
328
+ "status_code": 200,
329
+ "duration_ms": 1523,
330
+ "slow_request": true,
331
+ "slow_threshold_ms": 1000,
332
+ "level": "warning"
333
+ }
334
+ ```
335
+
336
+ **Error with Sanitized Headers:**
337
+ ```json
338
+ {
339
+ "event": "request.end",
340
+ "request_id": "550e8400-e29b-41d4-a716-446655440002",
341
+ "method": "POST",
342
+ "path": "/api/login",
343
+ "status_code": 401,
344
+ "duration_ms": 12,
345
+ "request_headers": {
346
+ "authorization": "Bearer ***",
347
+ "content-type": "application/json"
348
+ },
349
+ "level": "warning"
350
+ }
351
+ ```
352
+
353
+ ---
354
+
355
+ ## 🎯 Why Structured Logging?
356
+
357
+ **Traditional logging challenges:**
358
+ - Plain text logs are hard to parse programmatically
359
+ - Difficult to filter and search in log aggregation tools
360
+ - Missing context makes debugging distributed systems challenging
361
+
362
+ **Structured logging benefits:**
363
+ - **Machine-readable**: JSON format for easy parsing by ELK, Datadog, Grafana Loki
364
+ - **Rich context**: Automatic correlation with request_id, user_id, transaction_id
365
+ - **Better filtering**: Query logs by any field (status_code, duration, user_id, etc.)
366
+ - **Observability**: Enhanced monitoring and alerting capabilities
367
+ - **Debugging**: Trace requests across microservices with distributed tracing support
368
+
369
+ **This package uses [structlog](https://www.structlog.org/)** - a powerful library that enhances Python's standard logging with better context management and flexible log formatting.
370
+
371
+
372
+ ---
373
+
374
+ ## 🛠️ Development
375
+
376
+ ### Prerequisites
377
+
378
+ Install [Poetry](https://python-poetry.org/docs/#installation) for dependency management:
379
+
380
+ ```bash
381
+ curl -sSL https://install.python-poetry.org | python3 -
382
+ ```
383
+
384
+ ### Setup
385
+
386
+ ```bash
387
+ # Install dependencies
388
+ poetry install --with dev
389
+
390
+ # Run tests with coverage
391
+ poetry run pytest -v --cov=ganicas_utils
392
+
393
+ # Run tests with detailed output
394
+ poetry run pytest -rs --cov=ganicas_utils -s
395
+
396
+ # Run pre-commit hooks
397
+ poetry run pre-commit run --all-files
398
+ ```
399
+
400
+ ### Running Tests
401
+
402
+ ```bash
403
+ # Run all tests
404
+ poetry run pytest
405
+
406
+ # Run specific test file
407
+ poetry run pytest tests/test_request_logging_middleware.py
408
+
409
+ # Run with coverage report
410
+ poetry run pytest --cov=ganicas_utils --cov-report=html
411
+ ```
412
+
413
+ ### Code Quality
414
+
415
+ This project uses:
416
+ - **pytest** for testing (99% coverage)
417
+ - **ruff** for linting and formatting
418
+ - **pre-commit** for automated checks
419
+
420
+ ---
421
+
422
+ ## 📄 License
423
+
424
+ Proprietary - Internal use only for Ganicas projects.
425
+
426
+ ---
427
+
428
+ ## 🤝 Contributing
429
+
430
+ This is an internal package. For questions or contributions, please contact the Ganicas development team.
431
+
432
+ ---
433
+
434
+ ## 📚 Additional Resources
435
+
436
+ - [structlog Documentation](https://www.structlog.org/en/stable/)
437
+ - [FastAPI Middleware Guide](https://fastapi.tiangolo.com/tutorial/middleware/)
438
+ - [Flask Middleware Guide](https://flask.palletsprojects.com/en/latest/api/#flask.Flask.wsgi_app)
439
+
440
+ ---
441
+
442
+ **Made with ❤️ by Ganicas Team**
443
+
@@ -0,0 +1,11 @@
1
+ ganicas_utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
2
+ ganicas_utils/config.py,sha256=-DUF1v0rlabzRGj8vt40-LQ4QwzbbKX9Ywetdu-yJW0,414
3
+ ganicas_utils/logging/__init__.py,sha256=CoAyxkRoqIXqIHDtVHEP3VUb83455eQ7mdMoZw_H9gw,197
4
+ ganicas_utils/logging/configuration.py,sha256=7IvdN7VBQ1gLSWXObEpCdV4L16mMK8KysiuyM4XM-1k,2068
5
+ ganicas_utils/logging/formatter.py,sha256=1wfbKrXkbh_Xdz6ipX4ut7xAAWGhSTR00OEE6S7QuEU,592
6
+ ganicas_utils/logging/logger.py,sha256=ablL50oJK2UNmHAJzUdWZKkz7VI99q32uP6cfLXUSVM,3340
7
+ ganicas_utils/logging/middlewares.py,sha256=2901vY8f1eImpZO-CqO7p_pCkir07IdDauZqFdSablw,9401
8
+ ganicas_utils/logging/utils.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
9
+ ganicas_package-0.2.0.dist-info/METADATA,sha256=U5cgWyIX8b8r2Tgc5kM5kzI4Nc7E6fll-HW59xWeT7E,12608
10
+ ganicas_package-0.2.0.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
11
+ ganicas_package-0.2.0.dist-info/RECORD,,
@@ -14,7 +14,7 @@ def get_default_logging_conf(log_level: str, formatter: str, formatter_std_lib:
14
14
  "format": "%(asctime)s %(levelname)s %(name)s %(message)s",
15
15
  },
16
16
  "json_formatter": {
17
- "()": "src.logging.formatter.LogFormatter",
17
+ "()": "ganicas_utils.logging.formatter.LogFormatter",
18
18
  },
19
19
  "plain_console": {
20
20
  "()": structlog.stdlib.ProcessorFormatter,
@@ -1,11 +1,13 @@
1
- from typing import Optional
2
1
  import logging
3
2
  import logging.config
3
+ from typing import Optional
4
+
4
5
  import structlog
5
6
  from structlog import contextvars
7
+ from structlog.dev import ConsoleRenderer
6
8
  from structlog.typing import EventDict
9
+
7
10
  from ganicas_utils.logging.configuration import get_default_logging_conf
8
- from structlog.dev import ConsoleRenderer
9
11
 
10
12
 
11
13
  class LoggingConfigurator:
@@ -1,10 +1,12 @@
1
1
  import uuid
2
+ from time import perf_counter
3
+ from typing import Any, Optional
2
4
 
3
5
  import structlog
4
- from fastapi import Request
5
- from starlette.middleware.base import BaseHTTPMiddleware
6
+ from starlette.types import Message, Receive, Scope, Send
6
7
 
7
8
 
9
+ SENSITIVE_HEADERS = {"authorization", "cookie", "set-cookie", "x-api-key"}
8
10
  class FlaskRequestContextMiddleware:
9
11
  """Middleware for Flask to add request context to structlog."""
10
12
 
@@ -27,22 +29,220 @@ class FlaskRequestContextMiddleware:
27
29
  return self.app(environ, start_response)
28
30
 
29
31
 
30
- async def add_request_context_fastapi(request: Request, call_next):
31
- """Middleware for FastAPI applications."""
32
- request_id = request.headers.get("x-amzn-trace-id")
33
- if not request_id:
34
- request_id = str(uuid.uuid4())
32
+ # --- ASGI Request Logging Middleware (FastAPI/Starlette) ---
35
33
 
36
- structlog.contextvars.bind_contextvars(
37
- request_id=request_id, request_method=request.method, request_path=str(request.url)
38
- )
39
- response = await call_next(request)
40
- structlog.contextvars.clear_contextvars()
41
- return response
42
34
 
35
+ def _headers_to_dict(raw_headers: list[tuple[bytes, bytes]]) -> dict[str, str]:
36
+ return {k.decode("latin-1").lower(): v.decode("latin-1") for k, v in (raw_headers or [])}
43
37
 
44
- class FastAPIRequestContextMiddleware(BaseHTTPMiddleware):
45
- """Middleware class for FastAPI using BaseHTTPMiddleware."""
46
38
 
47
- async def dispatch(self, request: Request, call_next):
48
- return await add_request_context_fastapi(request, call_next)
39
+ def _sanitize_headers(headers: dict[str, str]) -> dict[str, str]:
40
+ out: dict[str, str] = {}
41
+ for k, v in (headers or {}).items():
42
+ lk = k.lower()
43
+ if lk == "authorization":
44
+ # Preserve scheme (e.g., "Bearer ***" or "Basic ***")
45
+ parts = v.split(" ", 1)
46
+ if len(parts) == 2:
47
+ out[k] = f"{parts[0]} ***"
48
+ else:
49
+ out[k] = "***"
50
+ elif lk in SENSITIVE_HEADERS:
51
+ out[k] = "<redacted>"
52
+ else:
53
+ out[k] = v
54
+ return out
55
+
56
+
57
+ def _client_ip(scope: Scope, headers: dict[str, str]) -> Optional[str]:
58
+ xff = headers.get("x-forwarded-for")
59
+ if xff:
60
+ return xff.split(",")[0].strip()
61
+ client = scope.get("client")
62
+ return client[0] if client else None
63
+
64
+
65
+ def _safe_int(value: Optional[str], default: int = 0) -> int:
66
+ try:
67
+ if value is None:
68
+ return default
69
+ v = int(value)
70
+ return v if v >= 0 else default
71
+ except Exception:
72
+ return default
73
+
74
+
75
+ class RequestLoggingMiddleware:
76
+ """Lightweight ASGI middleware that logs each HTTP request/response.
77
+
78
+ - Uses structured logging with structlog
79
+ - Avoids reading request/response bodies (logs sizes instead)
80
+ - Distinguishes 2xx/3xx (info) vs 4xx (warning) vs 5xx (error)
81
+ - For errors, includes sanitized headers and traceback when exceptions occur
82
+ - Optionally propagates request_id to response headers
83
+ """
84
+
85
+ def __init__(
86
+ self,
87
+ app,
88
+ logger: Optional[structlog.BoundLoggerBase] = None,
89
+ slow_request_threshold_ms: Optional[int] = None,
90
+ propagate_request_id: bool = True,
91
+ skip_paths: Optional[set[str]] = None,
92
+ skip_prefixes: Optional[tuple[str, ...]] = None,
93
+ sample_2xx_rate: Optional[float] = None,
94
+ ) -> None:
95
+ self.app = app
96
+ self.logger = logger or structlog.get_logger("http")
97
+ self.slow_request_threshold_ms = slow_request_threshold_ms
98
+ self.propagate_request_id = propagate_request_id
99
+ self.skip_paths = skip_paths or {"/healthz", "/metrics"}
100
+ self.skip_prefixes = skip_prefixes or ("/metrics",)
101
+ self.sample_2xx_rate = sample_2xx_rate
102
+
103
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
104
+ if scope.get("type") != "http":
105
+ await self.app(scope, receive, send)
106
+ return
107
+
108
+ headers_raw: list[tuple[bytes, bytes]] = scope.get("headers") or []
109
+ req_headers = _headers_to_dict(headers_raw)
110
+
111
+ orig_traceparent = req_headers.get("traceparent")
112
+ request_id = (
113
+ orig_traceparent
114
+ or req_headers.get("x-request-id")
115
+ or req_headers.get("x-amzn-trace-id")
116
+ or str(uuid.uuid4())
117
+ )
118
+ method = scope.get("method", "UNKNOWN")
119
+ path = scope.get("path", "")
120
+ query_string = (scope.get("query_string") or b"").decode("latin-1")
121
+
122
+ # Bind core request context for correlation
123
+ structlog.contextvars.bind_contextvars(request_id=request_id, request_method=method, request_path=path)
124
+
125
+ client_ip = _client_ip(scope, req_headers)
126
+ user_agent = req_headers.get("user-agent")
127
+ req_content_length = _safe_int(req_headers.get("content-length"))
128
+ req_content_type = req_headers.get("content-type")
129
+
130
+ start = perf_counter()
131
+ resp: dict[str, Any] = {"status": None, "headers": [], "size": 0, "content_type": None}
132
+
133
+ async def send_wrapper(message: Message) -> None:
134
+ if message["type"] == "http.response.start":
135
+ resp["status"] = message.get("status")
136
+ hdrs = list(message.get("headers") or [])
137
+ resp["headers"] = hdrs
138
+ rh = _headers_to_dict(hdrs)
139
+ resp["content_type"] = rh.get("content-type")
140
+ if self.propagate_request_id:
141
+ names = {k.lower() for k, _ in hdrs}
142
+ if b"x-request-id" not in names:
143
+ hdrs.append((b"x-request-id", request_id.encode("latin-1")))
144
+ message["headers"] = hdrs
145
+ if orig_traceparent and b"traceparent" not in names:
146
+ hdrs.append((b"traceparent", orig_traceparent.encode("latin-1")))
147
+ message["headers"] = hdrs
148
+ elif message["type"] == "http.response.body":
149
+ body = message.get("body") or b""
150
+ resp["size"] += len(body)
151
+ await send(message)
152
+
153
+ log = self.logger.bind()
154
+
155
+ # Skip noisy paths entirely (health/metrics) and OPTIONS requests; also skip configured prefixes
156
+ skip_logging = (
157
+ method == "OPTIONS" or (path in self.skip_paths) or any(path.startswith(p) for p in self.skip_prefixes)
158
+ )
159
+ if not skip_logging:
160
+ log.info(
161
+ "request.start",
162
+ request_id=request_id,
163
+ method=method,
164
+ path=path,
165
+ query=query_string,
166
+ client_ip=client_ip,
167
+ user_agent=user_agent,
168
+ req_content_length=req_content_length,
169
+ req_content_type=req_content_type,
170
+ )
171
+
172
+ try:
173
+ await self.app(scope, receive, send_wrapper)
174
+ except Exception:
175
+ duration_ms = int((perf_counter() - start) * 1000)
176
+ # Log with traceback for unexpected exceptions
177
+ log.error(
178
+ "request.exception",
179
+ request_id=request_id,
180
+ method=method,
181
+ path=path,
182
+ query=query_string,
183
+ client_ip=client_ip,
184
+ user_agent=user_agent,
185
+ duration_ms=duration_ms,
186
+ req_content_length=req_content_length,
187
+ req_content_type=req_content_type,
188
+ status_code=500,
189
+ response_size=resp["size"],
190
+ exc_info=True,
191
+ request_headers=_sanitize_headers(req_headers),
192
+ )
193
+ # Ensure context is cleared even when exceptions occur
194
+ structlog.contextvars.clear_contextvars()
195
+ raise
196
+
197
+ duration_ms = int((perf_counter() - start) * 1000)
198
+ status = int(resp["status"] or 200)
199
+
200
+ # Sampling for successful responses
201
+ sampled_ok = True
202
+ if status < 400 and self.sample_2xx_rate is not None:
203
+ try:
204
+ import random
205
+
206
+ sampled_ok = random.random() < max(0.0, min(1.0, float(self.sample_2xx_rate)))
207
+ except Exception:
208
+ sampled_ok = True
209
+
210
+ is_slow = self.slow_request_threshold_ms is not None and duration_ms >= self.slow_request_threshold_ms
211
+
212
+ level = "info"
213
+ if status >= 500: # noqa: PLR2004
214
+ level = "error"
215
+ elif status >= 400 or ( # noqa: PLR2004
216
+ self.slow_request_threshold_ms is not None and duration_ms >= self.slow_request_threshold_ms
217
+ ):
218
+ level = "warning"
219
+
220
+ log_kwargs: dict[str, Any] = {
221
+ "request_id": request_id,
222
+ "method": method,
223
+ "path": path,
224
+ "query": query_string,
225
+ "client_ip": client_ip,
226
+ "user_agent": user_agent,
227
+ "status_code": status,
228
+ "duration_ms": duration_ms,
229
+ "response_size": int(resp["size"]),
230
+ "response_content_type": resp["content_type"],
231
+ "req_content_length": req_content_length,
232
+ "req_content_type": req_content_type,
233
+ "slow_request": is_slow,
234
+ "slow_threshold_ms": self.slow_request_threshold_ms,
235
+ }
236
+ if status >= 400: # noqa: PLR2004
237
+ log_kwargs["request_headers"] = _sanitize_headers(req_headers)
238
+ log_kwargs["response_headers"] = _sanitize_headers(_headers_to_dict(resp["headers"]))
239
+
240
+ if not skip_logging:
241
+ if level == "error":
242
+ log.error("request.end", **log_kwargs)
243
+ elif level == "warning":
244
+ log.warning("request.end", **log_kwargs)
245
+ elif sampled_ok:
246
+ log.info("request.end", **log_kwargs)
247
+
248
+ structlog.contextvars.clear_contextvars()
@@ -1,228 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: ganicas-package
3
- Version: 0.1.3
4
- Summary: Ganicas internal Python package for structured logging and utilities.
5
- Keywords: logging,utilities,internal-package
6
- Author: Ganicas
7
- Requires-Python: >=3.11,<4.0
8
- Classifier: Development Status :: 2 - Pre-Alpha
9
- Classifier: Intended Audience :: Developers
10
- Classifier: License :: Other/Proprietary License
11
- Classifier: Natural Language :: English
12
- Classifier: Programming Language :: Python :: 3
13
- Classifier: Programming Language :: Python :: 3.11
14
- Classifier: Programming Language :: Python :: 3.12
15
- Classifier: Programming Language :: Python :: 3.10
16
- Classifier: Programming Language :: Python :: 3.8
17
- Classifier: Programming Language :: Python :: 3.9
18
- Requires-Dist: fastapi (>=0.114.2,<0.116.0)
19
- Requires-Dist: flask (>=2.2.0,<3.0.0)
20
- Requires-Dist: httpx (>=0.28.1,<0.29.0)
21
- Requires-Dist: python-json-logger (>=3.2.1,<4.0.0)
22
- Requires-Dist: structlog (>=24.4.0,<25.0.0)
23
- Description-Content-Type: text/markdown
24
-
25
- # Ganicas Python Package
26
-
27
- ### Structlog
28
- Structlog is a powerful logging library for structured, context-aware logging.
29
- More details can be found in the [structlog](https://www.structlog.org/en/stable/).
30
-
31
- #### Example, basic structlog configuration
32
-
33
- instead of `logger = logging.getLogger(__name__)` it is `logger = structlog.get_logger(__name__)`
34
-
35
- ```python
36
- from src.logging import LoggingConfigurator
37
- from src.config import Config
38
- import structlog
39
-
40
- config = Config()
41
-
42
- LoggingConfigurator(
43
- service_name=config.APP_NAME,
44
- log_level='INFO',
45
- setup_logging_dict=True
46
- ).configure_structlog(
47
- formatter='plain_console',
48
- formatter_std_lib='plain_console'
49
- )
50
-
51
- logger = structlog.get_logger(__name__)
52
- logger.debug("This is a DEBUG log message", key_1="value_1", key_2="value_2", key_n="value_n")
53
- logger.info("This is an INFO log message", key_1="value_1", key_2="value_2", key_n="value_n")
54
- logger.warning("This is a WARNING log message", key_1="value_1", key_2="value_2", key_n="value_n")
55
- logger.error("This is an ERROR log message", key_1="value_1", key_2="value_2", key_n="value_n")
56
- logger.critical("This is a CRITICAL log message", key_1="value_1", key_2="value_2", key_n="value_n")
57
-
58
- try:
59
- 1 / 0
60
- except ZeroDivisionError:
61
- logger.exception("An EXCEPTION log with stack trace occurred", key_1="value_1", key_2="value_2")
62
-
63
-
64
- ```
65
- ![basic example](images/plain_console_logger.png)
66
-
67
-
68
- In production, you should aim for structured, machine-readable logs that can be easily ingested by log aggregation and monitoring tools like ELK (Elasticsearch, Logstash, Kibana), Datadog, or Prometheus:
69
-
70
- ```python
71
- from src.logging import LoggingConfigurator
72
- from ssrc.config import Config
73
- import structlog
74
-
75
- config = Config()
76
-
77
- LoggingConfigurator(
78
- service_name=config.APP_NAME,
79
- log_level='INFO',
80
- setup_logging_dict=True
81
- ).configure_structlog(
82
- formatter='json_formatter',
83
- formatter_std_lib='json_formatter'
84
- )
85
-
86
- logger = structlog.get_logger(__name__)
87
- logger.debug("This is a DEBUG log message", key_1="value_1", key_2="value_2", key_n="value_n")
88
- logger.info("This is an INFO log message", key_1="value_1", key_2="value_2", key_n="value_n")
89
- logger.warning("This is a WARNING log message", key_1="value_1", key_2="value_2", key_n="value_n")
90
- logger.error("This is an ERROR log message", key_1="value_1", key_2="value_2", key_n="value_n")
91
- logger.critical("This is a CRITICAL log message", key_1="value_1", key_2="value_2", key_n="value_n")
92
-
93
- try:
94
- 1 / 0
95
- except ZeroDivisionError:
96
- logger.exception("An EXCEPTION log with stack trace occurred", key_1="value_1", key_2="value_2")
97
- ```
98
-
99
- ![logger with different keys](images/json_logger.png)
100
-
101
-
102
- #### Using Middleware for Automatic Logging Context:
103
-
104
- The middleware adds request_id, IP, and user_id to every log during a request/response cycle.
105
- This middleware module provides logging context management for both Flask and FastAPI applications using structlog.
106
-
107
- Flask Middleware (add_request_context_flask): Captures essential request data such as the request ID, method, and path, binding them to the structlog context for better traceability during the request lifecycle.
108
-
109
- FastAPI Middleware (add_request_context_fastapi): Captures similar request metadata, ensuring a request ID is present, generating one if absent.
110
- It binds the request context to structlog and clears it after the request completes.
111
-
112
- Class-Based Middleware (FastAPIRequestContextMiddleware): A reusable FastAPI middleware class that integrates with the BaseHTTPMiddleware and delegates the logging setup to the add_request_context_fastapi function.
113
-
114
- This setup ensures structured, consistent logging across both frameworks, improving traceability and debugging in distributed systems.
115
-
116
-
117
- This guide explains how to set up and use structlog for structured logging in a Flask application. The goal is to have a consistent and centralized logging setup that can be reused across the application.
118
- The logger is initialized once in the main application file (e.g., app.py).
119
-
120
- ```python
121
- import sys
122
- import uuid
123
- from flask import Flask, request
124
- from src.logging import LoggingConfigurator
125
- from src.logging.middlewares import add_request_context_flask
126
- from ssrc.config import Config
127
- import structlog
128
-
129
- config = Config()
130
-
131
- LoggingConfigurator(
132
- service_name=config.APP_NAME,
133
- log_level="INFO",
134
- setup_logging_dict=True,
135
- ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
136
-
137
- logger = structlog.get_logger(__name__)
138
-
139
- app = Flask(__name__)
140
-
141
- @app.before_request
142
- def set_logging_context():
143
- """Bind context for each request using the middleware."""
144
- add_request_context_flask()
145
- logger.info("Context set for request")
146
-
147
- with app.test_client() as client:
148
- dynamic_request_id = str(uuid.uuid4())
149
- client.get("/", headers={"X-User-Name": "John Doe", "X-Request-ID": dynamic_request_id})
150
- logger.info("Test client request sent", request_id=dynamic_request_id)
151
-
152
- ```
153
-
154
- ![logger with context flask](images/flask_logger_with_context.png)
155
-
156
- You can use the same logger instance across different modules by importing structlog directly.
157
- Example (services.py):
158
-
159
-
160
- ```python
161
- import structlog
162
-
163
- logger = structlog.get_logger(__name__)
164
- logger.info("Processing data started", data_size=100)
165
- ```
166
- Key Points:
167
-
168
- - Centralized Configuration: The logger is initialized once in app.py.
169
- - Consistent Usage: structlog.get_logger(__name__) is imported and used across all files.
170
- - Context Management: Context is managed using structlog.contextvars.bind_contextvars().
171
- - Structured Logging: The JSON formatter ensures logs are machine-readable.
172
-
173
- FastAPI:
174
-
175
- ```python
176
- import uuid
177
- from fastapi import FastAPI, Request
178
- from src.logging.middlewares import FastAPIRequestContextMiddleware
179
- import structlog
180
-
181
- config = Config()
182
-
183
- LoggingConfigurator(
184
- service_name=config.APP_NAME,
185
- log_level="INFO",
186
- setup_logging_dict=True,
187
- ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
188
-
189
- logger = structlog.get_logger(__name__)
190
- app = FastAPI()
191
- app.add_middleware(FastAPIRequestContextMiddleware)
192
-
193
- ```
194
- ![logger with context fastapi](images/fastapi_logger_with_context.png)
195
-
196
-
197
- Automatic injection of:
198
- - user_id
199
- - IP
200
- - request_id
201
- - request_method
202
-
203
-
204
- This a console view, in prod it will be json (using python json logging to have standard logging and structlog logging as close as possible)
205
-
206
-
207
- ### Why Use a Structured Logger?
208
- - Standard logging often outputs plain text logs, which can be challenging for log aggregation tools like EFK Stack or Grafana Loki to process effectively.
209
- - Structured logging outputs data in a machine-readable format (e.g., JSON), making it easier for log analysis tools to filter and process logs efficiently.
210
- - With structured logging, developers can filter logs by fields such as request_id, user_id, and transaction_id for better traceability across distributed systems.
211
- - The primary goal is to simplify debugging, enable better error tracking, and improve observability with enhanced log analysis capabilities.
212
- - Structured logs are designed to be consumed primarily by machines for monitoring and analytics, while still being readable for developers when needed.
213
- - This package leverages structlog, a library that enhances Python's standard logging by providing better context management and a flexible structure for log messages.
214
-
215
-
216
- # Development of this project
217
-
218
- Please install [poetry](https://python-poetry.org/docs/#installation) as this is the tool we use for releasing and development.
219
-
220
- poetry install && poetry run pytest -rs --cov=src -s
221
-
222
- To run tests inside docker:
223
-
224
- poetry install --with dev && poetry run pytest -rs --cov=src
225
-
226
- To run pre-commit:
227
- poetry run pre-commit run --all-files
228
-
@@ -1,11 +0,0 @@
1
- ganicas_utils/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
2
- ganicas_utils/config.py,sha256=-DUF1v0rlabzRGj8vt40-LQ4QwzbbKX9Ywetdu-yJW0,414
3
- ganicas_utils/logging/__init__.py,sha256=CoAyxkRoqIXqIHDtVHEP3VUb83455eQ7mdMoZw_H9gw,197
4
- ganicas_utils/logging/configuration.py,sha256=zx-n3V1arqGTCRGXWgD96YV5AesJ9AqiX4-npxlE3YY,2058
5
- ganicas_utils/logging/formatter.py,sha256=1wfbKrXkbh_Xdz6ipX4ut7xAAWGhSTR00OEE6S7QuEU,592
6
- ganicas_utils/logging/logger.py,sha256=O7vJdOnxP49IBHMy7y8sG0nl2RbtmxfhdGh3Gw0wmDI,3338
7
- ganicas_utils/logging/middlewares.py,sha256=FIEn-qGTmZtoLfJcJDGLAG1PprgxCacE0e4wBt1C44Y,1546
8
- ganicas_utils/logging/utils.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
9
- ganicas_package-0.1.3.dist-info/METADATA,sha256=AaRf0VqL6PRWTzoCz8R8sxGef-bVVF6lxZPJLYeyrvc,9024
10
- ganicas_package-0.1.3.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
11
- ganicas_package-0.1.3.dist-info/RECORD,,