ganicas-package 0.1.1__py3-none-any.whl

ganicas_package-0.1.1.dist-info/METADATA

@@ -0,0 +1,228 @@
+Metadata-Version: 2.1
+Name: ganicas-package
+Version: 0.1.1
+Summary: Ganicas internal Python package for structured logging and utilities.
+Keywords: logging,utilities,internal-package
+Author: Ganicas
+Requires-Python: >=3.8,<3.13
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: Other/Proprietary License
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: fastapi (>=0.114.2,<0.116.0)
+Requires-Dist: flask (>=2.2.0,<3.0.0)
+Requires-Dist: httpx (>=0.28.1,<0.29.0)
+Requires-Dist: python-json-logger (>=3.2.1,<4.0.0)
+Requires-Dist: structlog (>=24.4.0,<25.0.0)
+Description-Content-Type: text/markdown
+
+# Ganicas Python Package
+
+### Structlog
+Structlog is a powerful logging library for structured, context-aware logging.
+More details can be found in the [structlog](https://www.structlog.org/en/stable/).
+
+#### Example, basic structlog configuration
+
+instead of `logger = logging.getLogger(__name__)` it is `logger = structlog.get_logger(__name__)`
+
+```python
+    from src.logging import LoggingConfigurator
+    from src.config import Config
+    import structlog
+
+    config = Config()
+
+    LoggingConfigurator(
+        service_name=config.APP_NAME,
+        log_level='INFO',
+        setup_logging_dict=True
+    ).configure_structlog(
+        formatter='plain_console',
+        formatter_std_lib='plain_console'
+    )
+
+    logger = structlog.get_logger(__name__)
+    logger.debug("This is a DEBUG log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.info("This is an INFO log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.warning("This is a WARNING log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.error("This is an ERROR log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.critical("This is a CRITICAL log message", key_1="value_1", key_2="value_2", key_n="value_n")
+
+    try:
+        1 / 0
+    except ZeroDivisionError:
+        logger.exception("An EXCEPTION log with stack trace occurred", key_1="value_1", key_2="value_2")
+
+
+```
+![basic example](images/plain_console_logger.png)
+
+
+In production, you should aim for structured, machine-readable logs that can be easily ingested by log aggregation and monitoring tools like ELK (Elasticsearch, Logstash, Kibana), Datadog, or Prometheus:
+
+```python
+    from src.logging import LoggingConfigurator
+    from ssrc.config import Config
+    import structlog
+
+    config = Config()
+
+    LoggingConfigurator(
+        service_name=config.APP_NAME,
+        log_level='INFO',
+        setup_logging_dict=True
+    ).configure_structlog(
+        formatter='json_formatter',
+        formatter_std_lib='json_formatter'
+    )
+
+    logger = structlog.get_logger(__name__)
+    logger.debug("This is a DEBUG log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.info("This is an INFO log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.warning("This is a WARNING log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.error("This is an ERROR log message", key_1="value_1", key_2="value_2", key_n="value_n")
+    logger.critical("This is a CRITICAL log message", key_1="value_1", key_2="value_2", key_n="value_n")
+
+    try:
+        1 / 0
+    except ZeroDivisionError:
+        logger.exception("An EXCEPTION log with stack trace occurred", key_1="value_1", key_2="value_2")
+```
+
+![logger with different keys](images/json_logger.png)
+
+
+#### Using Middleware for Automatic Logging Context:
+
+The middleware adds request_id, IP, and user_id to every log during a request/response cycle.
+This middleware module provides logging context management for both Flask and FastAPI applications using structlog.
+
+Flask Middleware (add_request_context_flask): Captures essential request data such as the request ID, method, and path, binding them to the structlog context for better traceability during the request lifecycle.
+
+FastAPI Middleware (add_request_context_fastapi): Captures similar request metadata, ensuring a request ID is present, generating one if absent.
+It binds the request context to structlog and clears it after the request completes.
+
+Class-Based Middleware (FastAPIRequestContextMiddleware): A reusable FastAPI middleware class that integrates with the BaseHTTPMiddleware and delegates the logging setup to the add_request_context_fastapi function.
+
+This setup ensures structured, consistent logging across both frameworks, improving traceability and debugging in distributed systems.
+
+
+This guide explains how to set up and use structlog for structured logging in a Flask application. The goal is to have a consistent and centralized logging setup that can be reused across the application.
+The logger is initialized once in the main application file (e.g., app.py).
+
+```python
+    import sys
+    import uuid
+    from flask import Flask, request
+    from src.logging import LoggingConfigurator
+    from src.logging.middlewares import add_request_context_flask
+    from ssrc.config import Config
+    import structlog
+
+    config = Config()
+
+    LoggingConfigurator(
+        service_name=config.APP_NAME,
+        log_level="INFO",
+        setup_logging_dict=True,
+    ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
+
+    logger = structlog.get_logger(__name__)
+
+    app = Flask(__name__)
+
+    @app.before_request
+    def set_logging_context():
+        """Bind context for each request using the middleware."""
+        add_request_context_flask()
+        logger.info("Context set for request")
+
+    with app.test_client() as client:
+        dynamic_request_id = str(uuid.uuid4())
+        client.get("/", headers={"X-User-Name": "John Doe", "X-Request-ID": dynamic_request_id})
+        logger.info("Test client request sent", request_id=dynamic_request_id)
+
+```
+
+![logger with context flask](images/flask_logger_with_context.png)
+
+You can use the same logger instance across different modules by importing structlog directly.
+Example (services.py):
+
+
+```python
+    import structlog
+
+    logger = structlog.get_logger(__name__)
+    logger.info("Processing data started", data_size=100)
+```
+Key Points:
+
+- Centralized Configuration: The logger is initialized once in app.py.
+- Consistent Usage: structlog.get_logger(__name__) is imported and used across all files.
+- Context Management: Context is managed using structlog.contextvars.bind_contextvars().
+- Structured Logging: The JSON formatter ensures logs are machine-readable.
+
+FastAPI:
+
+```python
+    import uuid
+    from fastapi import FastAPI, Request
+    from src.logging.middlewares import FastAPIRequestContextMiddleware
+    import structlog
+
+    config = Config()
+
+    LoggingConfigurator(
+        service_name=config.APP_NAME,
+        log_level="INFO",
+        setup_logging_dict=True,
+    ).configure_structlog(formatter='json_formatter', formatter_std_lib='json_formatter')
+
+    logger = structlog.get_logger(__name__)
+    app = FastAPI()
+    app.add_middleware(FastAPIRequestContextMiddleware)
+
+```
+![logger with context fastapi](images/fastapi_logger_with_context.png)
+
+
+Automatic injection of:
+-   user_id
+-   IP
+-   request_id
+-  request_method
+
+
+This a console view, in prod it will be json (using python json logging to have standard logging and structlog logging as close as possible)
+
+
+### Why Use a Structured Logger?
+-   Standard logging often outputs plain text logs, which can be challenging for log aggregation tools like EFK Stack or Grafana Loki to process effectively.
+-   Structured logging outputs data in a machine-readable format (e.g., JSON), making it easier for log analysis tools to filter and process logs efficiently.
+-   With structured logging, developers can filter logs by fields such as request_id, user_id, and transaction_id for better traceability across distributed systems.
+-   The primary goal is to simplify debugging, enable better error tracking, and improve observability with enhanced log analysis capabilities.
+-   Structured logs are designed to be consumed primarily by machines for monitoring and analytics, while still being readable for developers when needed.
+-   This package leverages structlog, a library that enhances Python's standard logging by providing better context management and a flexible structure for log messages.
+
+
+# Development of this project
+
+Please install [poetry](https://python-poetry.org/docs/#installation) as this is the tool we use for releasing and development.
+
+    poetry install && poetry run pytest -rs --cov=src -s
+
+To run tests inside docker:
+
+    poetry install --with dev && poetry run pytest -rs --cov=src
+
+To run pre-commit:
+    poetry run pre-commit run --all-files
+

ganicas_package-0.1.1.dist-info/RECORD

@@ -0,0 +1,11 @@
+src/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+src/config.py,sha256=-DUF1v0rlabzRGj8vt40-LQ4QwzbbKX9Ywetdu-yJW0,414
+src/logging/__init__.py,sha256=DBfIpUMPGMC4LnK67bCOVPU6HyehUPMs7ayrouilrwg,177
+src/logging/configuration.py,sha256=Md02lL1-Oz2Qn3FOCe4OHr2cXwvdcTLrR0SW7lE4wkE,2048
+src/logging/formatter.py,sha256=nS3BJcg_evl3q-A2yyTSUS-iXaWqnxyAKwvAfO7CKUQ,582
+src/logging/logger.py,sha256=IPTyqwnSt2R621qTYmuaHhx4wEe9rDX9FM2Ddo1Zc1Q,3328
+src/logging/middlewares.py,sha256=FIEn-qGTmZtoLfJcJDGLAG1PprgxCacE0e4wBt1C44Y,1546
+src/logging/utils.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ganicas_package-0.1.1.dist-info/METADATA,sha256=Y2tA4P0l4vgemZsxHCudA7ztyUUsPyj48mQrFHjS3vo,9024
+ganicas_package-0.1.1.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
+ganicas_package-0.1.1.dist-info/RECORD,,

ganicas_package-0.1.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 1.9.0
+Root-Is-Purelib: true
+Tag: py3-none-any

src/config.py

@@ -0,0 +1,17 @@
+import os
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Config:
+    APP_NAME: str = field(init=False)
+    ENVIRONMENT: str = field(init=False)
+    LOG_LEVEL: str = field(init=False)
+
+    def __post_init__(self):
+        self.APP_NAME = os.getenv("APP_NAME", "App")
+        self.ENVIRONMENT = os.getenv("ENVIRONMENT", "dev")
+        self.LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
+
+
+config = Config()

src/logging/__init__.py

@@ -0,0 +1,4 @@
+from src.logging.configuration import get_default_logging_conf
+from src.logging.logger import LoggingConfigurator
+
+__all__ = ["LoggingConfigurator", "get_default_logging_conf"]

src/logging/configuration.py

@@ -0,0 +1,69 @@
+from typing import Any
+
+import structlog
+
+from src.config import Config
+
+
+def get_default_logging_conf(log_level: str, formatter: str, formatter_std_lib: str) -> dict[str, Any]:
+    config_instance = Config()
+    app_name = config_instance.APP_NAME
+
+    formatters = {
+        "verbose": {
+            "format": "%(asctime)s %(levelname)s %(name)s %(message)s",
+        },
+        "json_formatter": {
+            "()": "src.logging.formatter.LogFormatter",
+        },
+        "plain_console": {
+            "()": structlog.stdlib.ProcessorFormatter,
+            "processor": structlog.dev.ConsoleRenderer(event_key="message"),
+        },
+        "plain_console_std_lib": {
+            "()": "logging.Formatter",
+            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        },
+        "key_value": {
+            "()": structlog.stdlib.ProcessorFormatter,
+            "processor": structlog.processors.KeyValueRenderer(
+                key_order=["microservice", "timestamp", "level", "event", "logger"]
+            ),
+        },
+    }
+
+    if formatter not in formatters or formatter_std_lib not in formatters:
+        raise NotImplementedError("formatter not supported")
+
+    config: dict[str, Any] = {
+        "version": 1,
+        "disable_existing_loggers": False,
+        "formatters": formatters,
+        "handlers": {
+            "console": {
+                "level": log_level,
+                "class": "logging.StreamHandler",
+                "formatter": formatter,
+            },
+            "console_std_lib": {
+                "level": log_level,
+                "class": "logging.StreamHandler",
+                "formatter": formatter_std_lib,
+            },
+        },
+        "loggers": {
+            "": {
+                "level": log_level,
+                "handlers": ["console_std_lib"],
+                "propagate": False,
+            },
+        },
+    }
+
+    config["loggers"][app_name] = {
+        "level": log_level,
+        "handlers": ["console"],
+        "propagate": False,
+    }
+
+    return config

src/logging/formatter.py

@@ -0,0 +1,15 @@
+from datetime import datetime, timezone
+
+from pythonjsonlogger import jsonlogger
+
+from src.config import config
+
+
+class LogFormatter(jsonlogger.JsonFormatter):
+    def add_fields(self, log_record, record, message_dict):
+        super().add_fields(log_record, record, message_dict)
+        log_record["microservice"] = config.APP_NAME
+        if not log_record.get("timestamp"):
+            now = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+            log_record["timestamp"] = now
+        log_record["level"] = log_record.get("level", record.levelname).upper()

src/logging/logger.py

@@ -0,0 +1,90 @@
+from typing import Optional
+import logging
+import logging.config
+import structlog
+from structlog import contextvars
+from structlog.typing import EventDict
+from src.logging.configuration import get_default_logging_conf
+from structlog.dev import ConsoleRenderer
+
+
+class LoggingConfigurator:
+    def __init__(
+        self,
+        service_name: str,
+        log_level: str = "INFO",
+        config: Optional[dict] = None,
+        setup_logging_dict: bool = False,
+    ):
+        self.service_name = service_name
+        self.log_level = log_level.upper()
+        self.config = config
+        self.setup_logging_dict = setup_logging_dict
+
+    @staticmethod
+    def add_logger_name(logger: logging.Logger, method_name: str, event_dict: EventDict) -> EventDict:
+        """
+        Adds the logger name to the log event dictionary, keeping compatibility with structlog's
+        processor signature. The `method_name` parameter is included to maintain compatibility with structlog's
+        processor signature but is not used directly in this method.
+        """
+
+        record = event_dict.get("_record")
+        event_dict["name"] = record.name if record else logger.name
+        return event_dict
+
+    def get_base_processors(self) -> list:
+        """Returns the base processors for structlog, common to all formats."""
+        return [
+            contextvars.merge_contextvars,
+            self.add_logger_name,
+            structlog.stdlib.add_log_level,
+            structlog.stdlib.filter_by_level,
+            structlog.processors.TimeStamper(fmt="iso"),
+            structlog.stdlib.PositionalArgumentsFormatter(),
+            structlog.processors.StackInfoRenderer(),
+            structlog.dev.set_exc_info,
+            structlog.processors.format_exc_info,
+            structlog.processors.UnicodeDecoder(),
+            structlog.processors.ExceptionPrettyPrinter(),
+            structlog.stdlib.ExtraAdder(),
+            structlog.processors.EventRenamer(to="message"),
+        ]
+
+    def get_processors(self, formatter: str = "json_formatter") -> list:
+        """
+        Returns processors based on the formatter type.
+        """
+        processors = self.get_base_processors()
+
+        if formatter == "plain_console":
+            processors.append(ConsoleRenderer(colors=True))
+        else:
+            processors.append(structlog.stdlib.ProcessorFormatter.wrap_for_formatter)
+
+        return processors
+
+    def configure_structlog(
+        self,
+        custom_processors: Optional[list] = None,
+        formatter: str = "json_formatter",
+        formatter_std_lib: str = "json_formatter",
+    ) -> None:
+        """Configures the structlog and standard Python logging."""
+        if self.setup_logging_dict:
+            logger_init_config = self.config or get_default_logging_conf(
+                log_level=self.log_level,
+                formatter=formatter,
+                formatter_std_lib=formatter_std_lib,
+            )
+
+            logging.config.dictConfig(logger_init_config)
+
+        processors = custom_processors or self.get_processors(formatter=formatter)
+
+        structlog.configure(
+            processors=processors,
+            logger_factory=structlog.stdlib.LoggerFactory(),
+            wrapper_class=structlog.stdlib.BoundLogger,
+            cache_logger_on_first_use=True,
+        )

src/logging/middlewares.py

@@ -0,0 +1,48 @@
+import uuid
+
+import structlog
+from fastapi import Request
+from starlette.middleware.base import BaseHTTPMiddleware
+
+
+class FlaskRequestContextMiddleware:
+    """Middleware for Flask to add request context to structlog."""
+
+    def __init__(self, app):
+        self.app = app
+
+    def __call__(self, environ, start_response):
+        """Middleware logic executed for every HTTP request."""
+
+        request_id = environ.get("HTTP_X_REQUEST_ID", str(uuid.uuid4()))
+        request_method = environ.get("REQUEST_METHOD", "UNKNOWN")
+        request_path = environ.get("PATH_INFO", "")
+
+        structlog.contextvars.bind_contextvars(
+            request_id=request_id,
+            request_method=request_method,
+            request_path=request_path,
+        )
+
+        return self.app(environ, start_response)
+
+
+async def add_request_context_fastapi(request: Request, call_next):
+    """Middleware for FastAPI applications."""
+    request_id = request.headers.get("x-amzn-trace-id")
+    if not request_id:
+        request_id = str(uuid.uuid4())
+
+    structlog.contextvars.bind_contextvars(
+        request_id=request_id, request_method=request.method, request_path=str(request.url)
+    )
+    response = await call_next(request)
+    structlog.contextvars.clear_contextvars()
+    return response
+
+
+class FastAPIRequestContextMiddleware(BaseHTTPMiddleware):
+    """Middleware class for FastAPI using BaseHTTPMiddleware."""
+
+    async def dispatch(self, request: Request, call_next):
+        return await add_request_context_fastapi(request, call_next)