structlog-throttling 1.1.0__tar.gz

structlog_throttling-1.1.0/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: structlog-throttling
+Version: 1.1.0
+Summary: Log throttling utilities for structlog.
+Keywords: structlog,throttling,throttle
+License-Expression: MIT OR Apache-2.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: System :: Logging
+Classifier: Typing :: Typed
+Requires-Dist: structlog>=25.5.0
+Requires-Python: >=3.9
+Project-URL: Codeberg, https://codeberg.org/tomasfarias/structlog-throttling
+Description-Content-Type: text/markdown
+
+# *structlog-throttling*: Throttling for *[structlog](https://www.structlog.org/)* loggers
+
+<a href="https://pypi.org/project/structlog-throttling/"><img src="https://img.shields.io/pypi/pyversions/structlog-throttling.svg" alt="Supported Python versions from PyPI." /></a>
+
+Logging offers a trade-off between visibility and performance. A particularly high performance cost can be incurred when logging in each iteration of a loop, common [hot spots](https://en.wikipedia.org/wiki/Hot_spot_%28computer_programming%29) in most programs. A solution to this problem is to space out the log calls such that they only happen every some time instead of on every iteration of the loop. By tweaking the time in between log calls we can move within the visibility-performance trade-off.
+
+*structlog-throttling* brings this solution to *[structlog](https://www.structlog.org/)* in the form of processors to throttle log calls based on time, or call count.
+
+## Getting started
+
+### Installation
+
+Install *structlog-throttling* from PyPI:
+
+```sh
+pip install structlog-throttling
+```
+
+### Configure
+
+When configuring *structlog*, use one of the processors offered by *structlog-throttling*. I recommend putting the processor close to the beginning of your processor chain, to avoid processing logs that will ultimately be dropped:
+
+```python
+import structlog
+from structlog_throttling.processors import LogTimeThrottler
+
+
+structlog.configure(
+    processors=[
+        # Logs with the same 'event' will only be allowed through every 5 seconds.
+        LogTimeThrottler("event", every_seconds=5),
+        ...
+    ]
+)
+```
+
+## Examples
+
+Throttle logs based on log level every 5 seconds:
+
+```python
+import structlog
+from structlog_throttling.processors import LogTimeThrottler
+
+
+structlog.configure(
+    processors=[
+        structlog.processors.add_log_level,
+        LogTimeThrottler("level", every_seconds=5),
+        ...
+    ],
+)
+```

structlog_throttling-1.1.0/README.md

@@ -0,0 +1,53 @@
+# *structlog-throttling*: Throttling for *[structlog](https://www.structlog.org/)* loggers
+
+<a href="https://pypi.org/project/structlog-throttling/"><img src="https://img.shields.io/pypi/pyversions/structlog-throttling.svg" alt="Supported Python versions from PyPI." /></a>
+
+Logging offers a trade-off between visibility and performance. A particularly high performance cost can be incurred when logging in each iteration of a loop, common [hot spots](https://en.wikipedia.org/wiki/Hot_spot_%28computer_programming%29) in most programs. A solution to this problem is to space out the log calls such that they only happen every some time instead of on every iteration of the loop. By tweaking the time in between log calls we can move within the visibility-performance trade-off.
+
+*structlog-throttling* brings this solution to *[structlog](https://www.structlog.org/)* in the form of processors to throttle log calls based on time, or call count.
+
+## Getting started
+
+### Installation
+
+Install *structlog-throttling* from PyPI:
+
+```sh
+pip install structlog-throttling
+```
+
+### Configure
+
+When configuring *structlog*, use one of the processors offered by *structlog-throttling*. I recommend putting the processor close to the beginning of your processor chain, to avoid processing logs that will ultimately be dropped:
+
+```python
+import structlog
+from structlog_throttling.processors import LogTimeThrottler
+
+
+structlog.configure(
+    processors=[
+        # Logs with the same 'event' will only be allowed through every 5 seconds.
+        LogTimeThrottler("event", every_seconds=5),
+        ...
+    ]
+)
+```
+
+## Examples
+
+Throttle logs based on log level every 5 seconds:
+
+```python
+import structlog
+from structlog_throttling.processors import LogTimeThrottler
+
+
+structlog.configure(
+    processors=[
+        structlog.processors.add_log_level,
+        LogTimeThrottler("level", every_seconds=5),
+        ...
+    ],
+)
+```

structlog_throttling-1.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["uv_build>=0.9.8,<0.10.0"]
+build-backend = "uv_build"
+
+[project]
+name = "structlog-throttling"
+version = "1.1.0"
+description = "Log throttling utilities for structlog."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT OR Apache-2.0"
+keywords = ["structlog", "throttling", "throttle"]
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "License :: OSI Approved :: Apache Software License",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Topic :: System :: Logging",
+  "Typing :: Typed",
+]
+dependencies = [
+    "structlog>=25.5.0",
+]
+
+[project.urls]
+Codeberg = "https://codeberg.org/tomasfarias/structlog-throttling"
+
+[tool.pytest.ini_options]
+testpaths = "tests"
+
+[tool.ruff]
+src = ["src", "tests"]
+line-length = 88
+
+[dependency-groups]
+dev = [
+    "pytest>=8.4.2",
+]

structlog_throttling-1.1.0/src/structlog_throttling/__init__.py

@@ -0,0 +1,3 @@
+__title__ = "structlog-throttling"
+__author__ = "Tomás Farías Santana"
+__copyright__ = "Copyright (c) 2025 Tomás Farías Santana"

structlog_throttling-1.1.0/src/structlog_throttling/processors.py

@@ -0,0 +1,83 @@
+from __future__ import annotations
+
+import logging
+
+from structlog import DropEvent
+from structlog.typing import EventDict
+
+from .throttlers import CountThrottler, ThrottlerProtocol, TimeThrottler
+
+__all__ = [
+    "LogThrottler",
+    "LogTimeThrottler",
+    "CountThrottler",
+]
+
+
+class LogThrottler:
+    """Drop logs when throttled based on *throttler*.
+
+    This should generally be close to the top of your processor chain so that a log that
+    will ultimately be throttled is not processed further.
+
+    Args:
+        key: Unique key in the *event_dict* to determine if log should be throttled.
+        throttler:
+            A ``ThrottlerProtocol`` implementation to decide if *key should be
+            throttled.
+    """
+
+    def __init__(
+        self,
+        key: str,
+        throttler: ThrottlerProtocol,
+    ):
+        self.key = key
+        self.throttler = throttler
+
+    def __call__(self, _: logging.Logger, __: str, event_dict: EventDict) -> EventDict:
+        if self.throttler.is_throttled(event_dict[self.key]):
+            raise DropEvent
+        return event_dict
+
+
+class LogTimeThrottler:
+    """Drop logs when throttled based on time in between calls.
+
+    This is a convinience class to initialize a ``LogThrottler`` with a
+    ``TimeThrottler``.
+
+    Args:
+        key: Unique key in the *event_dict* to determine if log should be throttled.
+        every_seconds: How long to throttle logs for, in seconds.
+    """
+
+    def __init__(self, key: str, every_seconds: int | float) -> None:
+        self.key = key
+        self.throttler = TimeThrottler(every_seconds)
+
+    def __call__(self, _: logging.Logger, __: str, event_dict: EventDict) -> EventDict:
+        if self.throttler.is_throttled(event_dict[self.key]):
+            raise DropEvent
+        return event_dict
+
+
+class LogCountThrottler:
+    """Drop logs when throttled based on the number of times *key* was in a log call.
+
+    This is a convinience class to initialize a ``LogThrottler`` with a
+    ``CountThrottler``.
+
+    Args:
+        key: Unique key in the *event_dict* to determine if log should be throttled.
+        every: How frequently to throttle logs.
+    """
+
+    def __init__(self, key: str, every: int) -> None:
+        self.key = key
+        self.throttler = CountThrottler(every)
+
+    def __call__(self, _: logging.Logger, __: str, event_dict: EventDict) -> EventDict:
+        if self.throttler.is_throttled(event_dict[self.key]):
+            raise DropEvent
+        return event_dict

structlog_throttling-1.1.0/src/structlog_throttling/throttlers.py

@@ -0,0 +1,145 @@
+from __future__ import annotations
+
+import time
+import typing
+import weakref
+
+
+class ThrottlerProtocol(typing.Protocol):
+    def is_throttled(self, key: str) -> bool: ...
+
+
+class _Hashable(typing.Protocol):
+    def __hash__(self) -> int: ...
+
+
+class _Link:
+    """A link in a doubly-linked list"""
+
+    __slots__ = "at", "previous", "next", "__weakref__"
+    previous: "_Link" | None
+    next: "_Link" | None
+    at: float | None
+
+
+class TimeThrottler:
+    """A throttler for time-based throttling.
+
+    The intuition is that if we determine that a particular key is no longer throttled,
+    then it follows that any key that came before it (i.e. any key that was throttled
+    with an earlier timestamp) is also no longer throttled. Thus it is possible to save
+    memory by clearing multiple keys simultaneously.
+
+    This is achieved by maintaining two data structures:
+    * A doubly-linked list of ``_Link``.
+    * A ``weakref.WeakValueDictionary`` mapping keys to links in the list.
+
+    In each ``_Link`` of the doubly-linked list we store:
+    * The timestamp observed when a ``key`` is throttled.
+    * A weak reference to the ``next`` ``_Link``, which will contain the registered
+        timestamp for the next ``key`` that was throttled.
+    * A strong reference to the ``previous`` ``_Link``, which will contain the
+        registered timestamp for the ``key`` that was just previously throttled.
+
+    The idea is that there is only one strong reference to each link: In the link that
+    comes next in the list. This allows preserving memory by dropping multiple keys from
+    the list simultaneously by dropping a single link in the list. By dropping a single
+    link, we will drop the only strong reference to the previous link, thus it will also
+    be garbage collected, and this will subsequently drop the only reference to the
+    previous link of the previous link, which will also be garbage collected, and so on.
+
+    This design was inspired by ``collections.OrderedDict``.
+    """
+
+    def __init__(self, every_seconds: int | float) -> None:
+        self.every = every_seconds
+
+        self._last: _Link | None = None
+        self._indexes = weakref.WeakValueDictionary()
+
+    def is_throttled(self, key: _Hashable) -> bool:
+        """Determine whether *key* is throttled.
+
+        Examples:
+            >>> tt = TimeThrottler(every_seconds=1)
+            >>> tt.is_throttled("event")
+            False
+            >>> tt.is_throttled("event")
+            True
+            >>> tt.is_throttled("another-event")
+            False
+            >>> tt.is_throttled("another-event")
+            True
+            >>> time.sleep(1)
+            >>> tt.is_throttled("event")
+            False
+            >>> tt.is_throttled("another-event")
+            False
+        """
+        now = time.monotonic()
+
+        if key not in self._indexes:
+            new = _Link()
+            new.at = now
+            # Stores a weak reference
+            self._indexes[key] = new
+
+            if self._last:
+                # 'next' is a weak reference
+                self._last.next = self._indexes[key]
+                # 'previous' is not
+                new.previous = self._last
+
+            self._last = new
+
+            return False
+
+        link = self._indexes[key]
+        if (now - link.at) >= self.every:
+            if self._last == link:
+                # Unset '_last' otherwise we will still be holding a strong reference to
+                # 'link' and it won't be GC'd.
+                self._last = None
+
+            del link
+
+            return False
+
+        return True
+
+
+class CountThrottler:
+    """A throttler based on the count of times a key was seen."""
+
+    def __init__(self, every: int) -> None:
+        self.every = every
+
+        self._counts = {}
+
+    def is_throttled(self, key: _Hashable) -> bool:
+        """Determine whether *key* is throttled.
+
+        Examples:
+            >>> ct = CountThrottler(every=2)
+            >>> ct.is_throttled("event")
+            False
+            >>> ct.is_throttled("event")
+            True
+            >>> ct.is_throttled("event")
+            False
+        """
+        if key not in self._counts:
+            self._counts[key] = self.every
+
+        current = self._counts[key]
+        if current % self.every == 0:
+            should_throttle = False
+        else:
+            should_throttle = True
+
+        if current - 1 == 0:
+            del self._counts[key]
+        else:
+            self._counts[key] = current - 1
+
+        return should_throttle