victorialogs-handler 0.1.0a1__py3-none-any.whl

victorialogs_handler-0.1.0a1.dist-info/METADATA

@@ -0,0 +1,90 @@
+Metadata-Version: 2.4
+Name: victorialogs-handler
+Version: 0.1.0a1
+Summary: A log handler for VictoriaLogs.
+Author-email: Erik Kalkoken <kalkoken87@gmail.com>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-Expression: MIT
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Logging
+License-File: LICENSE
+Requires-Dist: orjson>3.10
+Requires-Dist: pdoc ; extra == "docs"
+Requires-Dist: coverage ; extra == "test"
+Project-URL: documentation, https://erikkalkoken.github.io/python-victorialogs-handler
+Project-URL: source, https://github.com/ErikKalkoken/python-victorialogs-handler
+Provides-Extra: docs
+Provides-Extra: test
+
+# victorialogs-handler
+
+A high-performance Python log handler for VictoriaLogs.
+
+[![release](https://img.shields.io/pypi/v/python-victorialogs-handler?label=release)](https://pypi.org/project/python-victorialogs-handler/)
+[![python](https://img.shields.io/pypi/pyversions/python-victorialogs-handler)](https://pypi.org/project/python-victorialogs-handler/)
+[![CI/CD](https://github.com/ErikKalkoken/python-victorialogs-handler/actions/workflows/cicd.yaml/badge.svg)](https://github.com/ErikKalkoken/python-victorialogs-handler/actions/workflows/cicd.yaml)
+[![codecov](https://codecov.io/gh/ErikKalkoken/python-victorialogs-handler/graph/badge.svg?token=2pPb3lid2k)](https://codecov.io/gh/ErikKalkoken/python-victorialogs-handler)
+[![license](https://img.shields.io/badge/license-MIT-green)](https://gitlab.com/ErikKalkoken/python-victorialogs-handler/-/blob/master/LICENSE)
+
+> [!IMPORTANT]
+> STATUS: In development. The API may still change.
+
+## Description
+
+**victorialogs-handler** is a high-performance Python log handler tailored for [VictoriaLogs](https://victoriametrics.com/products/victorialogs/). It integrates seamlessly with Python’s native logging module, allowing you to stream log events to a VictoriaLogs instance with minimal configuration.
+
+- Non-blocking: Logs are stored in a buffer and later processed in a background thread.
+- Hybrid trigger: Log processing is triggered by a ticker and/or when a size threshold is reached.
+- Batching: Multiple logs are combined into one request to the log server to minimize the amount of requests.
+- Complete logs: All fields of a log record are transmitted including exceptions and `extra` fields.
+- Highly customizable: The handler's behavior is highly customizable (see also [Documentation](#documentation))
+
+## Installation
+
+The handler can be installed with PIP from PyPI:
+
+```sh
+pip install victorialogs-handler
+```
+
+## Quick start
+
+> [!NOTE]
+> The script assumes that there is a VictoriaLogs server running
+> on the same system at the default URL: `http://localhost:9428`
+
+Here is a quick example on how to use the handler in your Python script:
+
+```python
+import logging
+
+from vlogs_handler import VictoriaLogsHandler
+
+# Create a custom logger with INFO level
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+# Add a handler for VictoriaLogs
+vlogs_handler = VictoriaLogsHandler()
+vlogs_handler.setLevel(logging.DEBUG)
+logger.addHandler(vlogs_handler)
+
+# Log example
+logger.info("This is an info message")
+```
+
+Please see the directory `/examples` for additional examples on how to use the handler.
+
+## Documentation
+
+The full documentation can be found here: [Documentation](https://erikkalkoken.github.io/python-victorialogs-handler/).
+

victorialogs_handler-0.1.0a1.dist-info/RECORD

@@ -0,0 +1,8 @@
+vlogs_handler/__init__.py,sha256=7rHydo4SrRUbMtjP2Ox4Uk0JwrK6qGpHViaja1_q-sQ,315
+vlogs_handler/handler.py,sha256=d7VM2k1vbwghteQaAjSwut78EwHoF0Jgmxr4xvs_2Yg,10080
+vlogs_handler/request.py,sha256=4njqG5uAOWlNJYOuhIEvX9t1EGWCci6HELChnJeKHAQ,1877
+vlogs_handler/technical.md,sha256=Jhcnu0lHMwRDvyEFrurBy5ZTsTWzbpuOZozU5qvxWl0,2671
+victorialogs_handler-0.1.0a1.dist-info/licenses/LICENSE,sha256=jvY6lHKeMpeOzbdm1LfDI5iujiRGfVKUW-sNhkYFfos,1080
+victorialogs_handler-0.1.0a1.dist-info/WHEEL,sha256=G2gURzTEtmeR8nrdXUJfNiB3VYVxigPQ-bEQujpNiNs,82
+victorialogs_handler-0.1.0a1.dist-info/METADATA,sha256=LK_wbKaVZZwz-rO5VRI8LdpmOLoCAoXlZVP3GbsT0Xg,3778
+victorialogs_handler-0.1.0a1.dist-info/RECORD,,

victorialogs_handler-0.1.0a1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: flit 3.12.0
+Root-Is-Purelib: true
+Tag: py3-none-any

victorialogs_handler-0.1.0a1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Erik Kalkoken
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

vlogs_handler/__init__.py

@@ -0,0 +1,16 @@
+"""
+This is the documentation for the VictoriaLogs Handler.
+
+# Readme
+.. include:: ../../README.md
+   :start-line: 2
+   :end-line: 59
+.. include:: technical.md
+"""
+
+from .handler import VictoriaLogsHandler  # noqa: F401
+
+__title__ = "VictoriaLogs Handler"
+__version__ = "0.1.0a1"
+
+__all__ = ["VictoriaLogsHandler"]

vlogs_handler/handler.py

@@ -0,0 +1,330 @@
+"""A module that provides the implementation of the vlogs logging handler."""
+
+import io
+import logging
+import os
+import queue
+import sys
+import threading
+import traceback
+import urllib.parse
+from typing import Callable, List, Optional, Tuple
+
+import orjson
+
+from . import request
+
+logger = logging.getLogger(__name__)
+
+_STANDARD_ATTRS: frozenset[str] = frozenset(
+    {
+        "args",
+        "asctime",
+        "created",
+        "exc_info",
+        "exc_text",
+        "filename",
+        "funcName",
+        "levelname",
+        "levelno",
+        "lineno",
+        "message",
+        "module",
+        "msecs",
+        "msg",
+        "name",
+        "pathname",
+        "process",
+        "processName",
+        "relativeCreated",
+        "stack_info",
+        "taskName",
+        "thread",
+        "threadName",
+    }
+)
+
+
+class VictoriaLogsHandler(logging.Handler):
+    """A handler class which dispatches logging records to a VictoriaLogs server.
+
+    Args:
+        batch_size: New logs are submitted immediately once this threshold is reached.
+        flush_interval: New logs are submitted every x seconds.
+        buffer_size: Maximum number of logs the buffer can hold.
+            If buffer_size <= 0, the size is unlimited (not recommended).
+            When the buffer is full any new logs will be discarded.
+            100 000 logs consume approx. 80-100 MB of RAM.
+            Tip: The buffer should be large enough to hold incoming logs
+            while the log server is down for regular maintenance.
+        chunk_size: Maximum number of logs send per request to the log server.
+        record_to_stream: A function that returns the value for the `stream`field
+            for a log record. The default will return the name of the top package.
+        request_timeout: Timeout when sending a request to the vlogs server in seconds.
+        start_worker: Whether to start the worker at initialization.
+            Alternatively, the worker can be started later by calling `start()`.
+        shutdown_timeout: Timeout when waiting for the worker to shut down.
+        url: URL of the vlogs server, e.g. `"http://localhost:9428"`
+    """
+
+    def __init__(
+        self,
+        batch_size: int = 125,
+        buffer_size: int = 100_000,
+        chunk_size: int = 1_000,
+        flush_interval: float = 5.0,
+        record_to_stream: Optional[Callable[[logging.LogRecord], str]] = None,
+        request_timeout: float = 3.0,
+        shutdown_timeout: float = 2.0,
+        start_worker: bool = True,
+        url: str = "http://localhost:9428",
+    ):
+        """Initializes the instance."""
+
+        super().__init__()
+
+        if batch_size < 1:
+            raise ValueError(f"batch_size must be >= 1: {batch_size}")
+
+        if chunk_size < 1:
+            raise ValueError(f"chunk_size must be >= 1: {chunk_size}")
+
+        if flush_interval < 0:
+            raise ValueError(f"flush_interval must be >= 0: {flush_interval}")
+
+        if record_to_stream and not callable(record_to_stream):
+            raise ValueError("record_to_stream must be a callable")
+
+        if request_timeout <= 0:
+            raise ValueError(f"request_timeout must be > 0: {request_timeout}")
+
+        if shutdown_timeout <= 0:
+            raise ValueError(f"shutdown_timeout must be > 0: {shutdown_timeout}")
+
+        if not request.is_url(url):
+            raise ValueError(f"url is not valid: {url}")
+
+        name = __package__
+        if not name:
+            raise RuntimeError("must run as module")
+
+        self.addFilter(_create_filter(name))
+
+        self._batch_size = int(batch_size)
+        self._buffer = queue.Queue(int(buffer_size))
+        self._chunk_size = int(chunk_size)
+        self._flush_interval = float(flush_interval)
+        self._record_to_stream = record_to_stream or _top_package_name
+        self._request_timeout = float(request_timeout)
+        self._shutdown_timeout = float(shutdown_timeout)
+        self._vlogs_url = (
+            urllib.parse.urljoin(url, "/insert/jsonline")
+            + "?"
+            + urllib.parse.urlencode(
+                {
+                    "_stream_fields": "stream",
+                    "_time_field": "timestamp",
+                    "_msg_field": "message",
+                }
+            )
+        )
+        self._worker_thread = threading.Thread(target=self._worker, daemon=True)
+
+        self._lock = threading.Lock()
+        self._worker_started = False
+        self._worker_run = threading.Event()
+        self._worker_shutdown = threading.Event()
+        self._added_count = 0
+
+        if start_worker:
+            self.start()
+
+    def close(self):
+        """Cleanup resources and flush the buffer."""
+        with self._lock:
+            if self._worker_shutdown.is_set():
+                return  # run shutdown once only
+
+            self._worker_shutdown.set()
+
+        if self._worker_started:
+            self._worker_run.set()
+            self._worker_thread.join(timeout=self._flush_interval)
+
+        try:
+            self.flush()
+        except Exception:
+            pass
+
+        try:
+            count = 0
+            stderr = sys.stderr.fileno()
+            while True:
+                try:
+                    log = self._buffer.get_nowait()
+                except queue.Empty:
+                    break
+
+                try:
+                    os.write(stderr, log + b"\n")  # print log to stderr
+                except Exception:
+                    break  # abort when writing to stderr no longer possible
+
+                count += 1
+
+            if count > 0:
+                print(
+                    f"Dumped {count} remaining logs to stderr during shutdown.",
+                    file=sys.stderr,
+                )
+        finally:
+            super().close()
+
+    def emit(self, record: logging.LogRecord) -> None:
+        """@private"""
+        try:
+            log = _serialize_log_to_json(record, self._record_to_stream)
+            self._buffer.put_nowait(log)
+
+        except Exception:
+            self.handleError(record)
+            return
+
+        with self._lock:
+            self._added_count += 1
+            if self._added_count > self._batch_size:
+                self._added_count = 0
+                self._worker_run.set()
+
+    def start(self):
+        """Start the worker. Is a no-op when the worker is already running."""
+        with self._lock:
+            if self._worker_started:
+                return
+
+            self._worker_started = True
+            self._worker_thread.start()
+
+        logger.debug("Worker started")
+
+    def _worker(self):
+        while not self._worker_shutdown.is_set():
+            self._worker_run.wait(timeout=self._flush_interval)
+            self._worker_run.clear()
+            self.flush()
+
+        logger.debug("Worker stopped")
+
+    def flush(self):
+        """Flush the buffer and send all logs to the log server."""
+        failed: List[bytes] = []
+        done = False
+
+        while not done:
+            logs: List[bytes] = []
+            while len(logs) < self._chunk_size:
+                try:
+                    logs.append(self._buffer.get_nowait())
+                except queue.Empty:
+                    done = True
+                    break
+
+            if logs:
+                ok = request.post_ndjson(
+                    url=self._vlogs_url, objs=logs, timeout=self._request_timeout
+                )
+                if not ok:
+                    failed += logs
+                    logger.warning("Failed transmitting %d logs to server", len(logs))
+                else:
+                    logger.debug("Completed transmitting %s logs to server", len(logs))
+
+        if not failed:
+            return
+
+        n = 0
+        for log in failed:
+            try:
+                self._buffer.put_nowait(log)
+                n += 1
+            except queue.Full:
+                logger.error(
+                    "Discarded %s logs after failed send because buffer is full",
+                    len(failed) - n,
+                )
+                break
+
+        logger.debug("Saved %d logs to buffer after failed send", n)
+
+
+def _serialize_log_to_json(
+    record: logging.LogRecord, record_to_stream: Callable[[logging.LogRecord], str]
+) -> bytes:
+    """Serialize a log record into a JSON object and return it."""
+    obj = {
+        "stream": record_to_stream(record),
+        "timestamp": record.created,
+        "level": record.levelname,
+        "logger": record.name,
+        "module": record.module,
+        "function": record.funcName,
+        "line_number": record.lineno,
+        "message": record.getMessage(),
+        "process_name": record.processName,
+        "process": record.process,
+        "thread_name": record.threadName,
+        "thread": record.thread,
+    }
+    if record.exc_info:
+        obj["exception_name"], obj["exception"] = _format_exception(record.exc_info)
+
+    for k, v in record.__dict__.items():
+        if k not in _STANDARD_ATTRS:
+            obj[k] = v
+
+    log = orjson.dumps(obj, default=str)
+    return log
+
+
+def _create_filter(name: str):
+    def filter_logic(record: logging.LogRecord) -> bool:
+        return not record.name.startswith(name)
+
+    f = logging.Filter()
+    f.filter = filter_logic
+    return f
+
+
+def _format_exception(ei) -> Tuple[str, str]:
+    """Format and return the name of the exception
+    and specified exception information as strings.
+
+    This default implementation just uses
+    traceback.print_exception()
+
+    Based on: logging.Formatter.formatException()
+    """
+    sio = io.StringIO()
+    tb = ei[2]
+    traceback.print_exception(ei[0], ei[1], tb, None, sio)
+    s = sio.getvalue()
+    sio.close()
+    if s[-1:] == "\n":
+        s = s[:-1]
+
+    name = ei[0].__name__ if ei[0] is not None else ""
+    return name, s
+
+
+def _top_package_name(record: logging.LogRecord) -> str:
+    """Return the top package name of a log."""
+    if record.name == "__main__":
+        return "(undefined)"
+
+    s = record.name.split(".")
+    if len(s) > 1:
+        return s[0]
+
+    return record.name
+    return record.name
+    return record.name

vlogs_handler/request.py

@@ -0,0 +1,69 @@
+"""A module that provides the functionality to send HTTP requests."""
+
+import logging
+import urllib.error
+import urllib.parse
+import urllib.request
+from typing import List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+def is_url(url: str) -> bool:
+    """Report whether a string represents a valid URL."""
+    try:
+        result = urllib.parse.urlparse(url)
+        return all([result.scheme, result.netloc])
+    except ValueError:
+        return False
+
+
+def post_ndjson(
+    *, url: str, objs: List[bytes], timeout: Optional[float] = None
+) -> bool:
+    """Send a POST request with the ndjson protocol
+    and report whether it was successful.
+
+    Args:
+        url: request URL
+        data: list of JSON objects to send
+        timeout: request timeout in seconds.
+            Settings it to None will disable the timeout.
+    """
+
+    data = b"\n".join(objs)
+    req = urllib.request.Request(url, data=data, method="POST")
+    req.add_header("Content-Type", "application/x-ndjson")
+
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            logger.debug("Submitted: %s %d %s", url, resp.status, resp.reason)
+            return True
+
+    except urllib.error.HTTPError as ex:
+        body = ex.read(4096).decode("utf-8")
+        logger.error(
+            "HTTP Error: %s",
+            ex.reason,
+            extra={
+                "url": ex.url,
+                "code": ex.code,
+                "reason": ex.reason,
+                "body": body,
+            },
+        )
+
+    except urllib.error.URLError as ex:
+        logger.error(
+            "URL Error: %s",
+            ex.reason,
+            extra={"url": url, "reason": ex.reason},
+        )
+
+    except TimeoutError:
+        logger.warning("Timed out", extra={"url": url})
+
+    except Exception:
+        logger.exception("general exception", extra={"url": url})
+
+    return False

vlogs_handler/technical.md

@@ -0,0 +1,55 @@
+# Technical overview
+
+This section gives a technical overview of how the vlogs handler works.
+
+## Process Flow
+
+The high-level process flow of the vlogs handler is as follows:
+
+1. When a a log event is received, it is converted into JSON and stored
+    in the buffer
+1. At the tick of an interval (e.g. 5 seconds) or when a threshold is reached
+    (e.g. 125 logs) a background worker starts the process of submitting logs
+    from the buffer to the log server
+1. Logs are combined into chunks (e.g. 1.000 logs per request)
+    and then submitted to the log server using vlog's the JSON Stream API
+
+## Failure behavior
+
+The failure behavior of the handler in key scenarios is as follows:
+
+- In case the submission to the log server fails
+    (e.g. the server is temporarily down) the logs will be stored
+    in the buffer for later retry
+- If the buffer is full, new log events will be discarded
+    and raise a exceptions (depending on logging configuration)
+- The handler will make a final attempt to submit remaining logs during shutdown.
+If that fails those logs will written to stderr.
+
+## LogRecord fields
+
+The following fields will be transferred for each log event. They are derived from Python's [LogRecord](https://docs.python.org/3/library/logging.html#logrecord-objects):
+
+Name | Description | Example | Optional
+-- | -- | -- | --
+`exception_name` | Name of the exception | `ZeroDivisionError` | yes
+`exception` | Full traceback of the exception | `Traceback ...` | yes
+`function` | Name of the function that emitted the log event | `my_function` | no
+`level` | Name of the level of the emitted log event | `INFO` | no
+`line_number` | Line number where the log event was emitted | `89` | no
+`logger` | Name of the related Python logger | `my_package.my_module` | no
+`message` | The logged message | 'This is a log entry' | no
+`stream` | This can be configured. By default this is name of the top-level Python package that emitted the log the event. | `my_package` | no
+`timestamp` | Timestamp of the log event, represented as fractional UNIX epoch | `1775081468.4308655` | no
+
+In addition any custom `extras' fields will be added as they are encountered.
+
+## VictoriaLogs special fields
+
+VictoriaLogs handles three fields in a special way:
+
+- `_msg`: The logged message. This is a mandatory field and is mapped to `message`.
+- `_time`: The timestamp of the log event. This field and is mapped to `timestamp`.
+- `_stream`: The source of a log event, which is used to group and filter logs. This field is mapped to `stream`.
+
+For more information please also see [VictoriaLogs Data model](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model).