analytics-platform 0.3.5__tar.gz → 0.3.6__tar.gz

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: analytics-platform
-Version: 0.3.5
+Version: 0.3.6
 Summary: Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard.
 Author: steviec87
 License-Expression: MIT
@@ -317,6 +317,8 @@ read, so no data leaves it.
 | pymongo | nothing, but **create the platform before your `MongoClient`**: pymongo only reports to clients made after the package is listening |
 | Any other DB-API driver (psycopg, psycopg2, mysqlclient, PyMySQL, sqlite3...) | `conn = platform.watch_database(psycopg.connect(...))`, and use `conn` as before |
 | ClickHouse: clickhouse-connect, clickhouse-driver | `client = platform.watch_database(clickhouse_connect.get_client(...))` (or a `clickhouse_driver.Client`), and use `client` as before |
+| Redis: redis-py (`Redis`, `RedisCluster`, sync or `redis.asyncio`) | `platform.watch_database(r)`: the client is watched in place, so keep using `r` |
+| Django's cache on Redis (Django's own `RedisCache`, or django-redis) | `platform.watch_database(caches["default"])`, once at start-up (in `AppConfig.ready()`) |
 
 A query outside any request or job is counted as `JOB (background)`. Each
 query is counted once: when SQLAlchemy or Django runs it over a watched
@@ -331,6 +333,23 @@ they answer. clickhouse-driver's `execute()` likewise. The streaming
 queries (`query_rows_stream()`, `execute_iter()`) aren't timed. Only the
 client you pass is wrapped; the library itself is left alone.
 
+Redis: each command is counted by its name and the shape of its key (`GET
+user:?:cart`); no other argument or value is read. A pipeline or transaction
+is one call (`PIPELINE GET user:?`, `MULTI SET cart:?`), timed as a whole.
+Commands that wait (`BLPOP`, `XREAD BLOCK`, subscribing) aren't timed, and
+nor are this package's own reads of your RQ queues. An error reply counts
+as failed; a nil reply doesn't.
+
+```python
+r = redis.Redis.from_url(REDIS_URL)
+platform.watch_database(r)
+
+# Django (settings: CACHES backend django.core.cache.backends.redis.RedisCache
+# or django_redis.cache.RedisCache), in your AppConfig.ready():
+from django.core.cache import caches
+platform.watch_database(caches["default"])
+```
+
 ## Outgoing calls
 
 Health → Server speed → Calls shows how long each route waited on other

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/README.md

@@ -291,6 +291,8 @@ read, so no data leaves it.
 | pymongo | nothing, but **create the platform before your `MongoClient`**: pymongo only reports to clients made after the package is listening |
 | Any other DB-API driver (psycopg, psycopg2, mysqlclient, PyMySQL, sqlite3...) | `conn = platform.watch_database(psycopg.connect(...))`, and use `conn` as before |
 | ClickHouse: clickhouse-connect, clickhouse-driver | `client = platform.watch_database(clickhouse_connect.get_client(...))` (or a `clickhouse_driver.Client`), and use `client` as before |
+| Redis: redis-py (`Redis`, `RedisCluster`, sync or `redis.asyncio`) | `platform.watch_database(r)`: the client is watched in place, so keep using `r` |
+| Django's cache on Redis (Django's own `RedisCache`, or django-redis) | `platform.watch_database(caches["default"])`, once at start-up (in `AppConfig.ready()`) |
 
 A query outside any request or job is counted as `JOB (background)`. Each
 query is counted once: when SQLAlchemy or Django runs it over a watched
@@ -305,6 +307,23 @@ they answer. clickhouse-driver's `execute()` likewise. The streaming
 queries (`query_rows_stream()`, `execute_iter()`) aren't timed. Only the
 client you pass is wrapped; the library itself is left alone.
 
+Redis: each command is counted by its name and the shape of its key (`GET
+user:?:cart`); no other argument or value is read. A pipeline or transaction
+is one call (`PIPELINE GET user:?`, `MULTI SET cart:?`), timed as a whole.
+Commands that wait (`BLPOP`, `XREAD BLOCK`, subscribing) aren't timed, and
+nor are this package's own reads of your RQ queues. An error reply counts
+as failed; a nil reply doesn't.
+
+```python
+r = redis.Redis.from_url(REDIS_URL)
+platform.watch_database(r)
+
+# Django (settings: CACHES backend django.core.cache.backends.redis.RedisCache
+# or django_redis.cache.RedisCache), in your AppConfig.ready():
+from django.core.cache import caches
+platform.watch_database(caches["default"])
+```
+
 ## Outgoing calls
 
 Health → Server speed → Calls shows how long each route waited on other

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform/__init__.py

@@ -342,12 +342,18 @@ class Platform:
         sqlite3...) or a ClickHouse client (clickhouse-connect,
         clickhouse-driver) whose queries are timed for Server speed: use what
         it returns in its place. db: "postgresql", "mysql"... when the driver
-        doesn't say. Django, SQLAlchemy and pymongo need nothing of this."""
+        doesn't say. Django, SQLAlchemy and pymongo need nothing of this.
+        A redis-py client (Redis, RedisCluster, sync or asyncio) or a Django
+        cache on Redis is watched in place and handed back as itself."""
         queries = self._queries()
         if queries is None:
             return connection
         from .queries import watch_database
-        return watch_database(connection, db)
+        try:
+            return watch_database(connection, db)
+        except Exception as error:
+            self.core.log("performance", "watch_database: its queries aren't timed:", error)
+            return connection
 
     def watch_http(self, client):
         """A requests Session, an httpx Client or AsyncClient, or an httpx

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform/core.py

@@ -23,7 +23,7 @@ import urllib.error
 import urllib.parse
 import urllib.request
 
-VERSION = "0.3.5"
+VERSION = "0.3.6"
 TIMEOUT = 5
 MIN_SYNC = 30
 RETRY = 60

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform/queries.py

@@ -15,6 +15,8 @@ Caught by, with nothing to set up beyond the platform itself:
   sqlite3...): conn = platform.watch_database(conn).
 - ClickHouse: a clickhouse-connect client (query, command, insert...) or a
   clickhouse-driver Client (execute): client = platform.watch_database(client).
+- Redis: a redis-py client (Redis, RedisCluster, sync or asyncio), watched in
+  place: platform.watch_database(r); a Django cache on Redis the same way.
 
 A query is timed at one layer only: the outer one (Django, SQLAlchemy) marks
 it as being timed, and a watched connection under it leaves it alone.
@@ -24,6 +26,7 @@ alongside, by call method and host (calls.py has the hooks), and go to
 POST /events/server-calls; calls=False leaves them out.
 """
 import contextvars
+import inspect
 import json
 import os
 import re
@@ -32,6 +35,7 @@ import threading
 import time
 import weakref
 from collections.abc import Mapping
+from contextlib import contextmanager
 from datetime import datetime, timezone
 from importlib.util import find_spec
 
@@ -41,7 +45,7 @@ MAX_STATEMENT = 1000
 OTHER_STATEMENT = "(other)"
 BACKGROUND = ("JOB", "(background)")
 # The databases a row may name; anything else is "other".
-DATABASES = frozenset(("postgresql", "mysql", "mariadb", "sqlite", "mssql", "oracle", "clickhouse", "mongodb", "other"))
+DATABASES = frozenset(("postgresql", "mysql", "mariadb", "sqlite", "mssql", "oracle", "clickhouse", "mongodb", "redis", "other"))
 # Past this many statements in one request, new ones are one "(other)".
 PER_REQUEST = 50
 # Past MAX_KEYS in a minute new statements fold into "(other)"; past
@@ -64,6 +68,9 @@ current_queries = contextvars.ContextVar("analytics_platform_queries", default=N
 # True while an outer layer (Django, SQLAlchemy) times a query, so a watched
 # connection under it doesn't count it again.
 _timing = contextvars.ContextVar("analytics_platform_query_timing", default=False)
+# True while the package reads Redis for itself (a queue's depth, queues.py):
+# a watched client it shares with the app leaves those commands out.
+_own_redis = contextvars.ContextVar("analytics_platform_own_redis", default=False)
 
 # ---- Normalizing: a port of expresscollect/Analytics/queryText.js ----
 
@@ -264,6 +271,142 @@ def _mongo_statement(text):
     return " ".join(part for part in (command, collection, shape) if part)
 
 
+# Redis: the command and its first key's shape, never an argument or value
+# (queries.md, "Redis"). A key keeps each ":" segment that is only letters
+# and "_" (32 at most); every other segment is "?", so IDs, slugs, emails and
+# hashes never leave the server: user:42:cart -> user:?:cart.
+_REDIS_WORD = re.compile(r"[A-Za-z][A-Za-z0-9_.|-]{0,39}\Z")
+_REDIS_SEGMENT = re.compile(r"[A-Za-z_]{1,32}\Z")
+# Commands whose second word is a subcommand (CLIENT SETNAME, XINFO STREAM).
+_REDIS_CONTAINERS = frozenset(("ACL", "CLIENT", "CLUSTER", "COMMAND", "CONFIG", "DEBUG", "FUNCTION", "LATENCY", "MEMORY",
+                               "MODULE", "OBJECT", "PUBSUB", "SCRIPT", "SLOWLOG", "XGROUP", "XINFO"))
+# Of those, the ones with a key after the subcommand.
+_REDIS_SUB_KEY = frozenset(("OBJECT", "MEMORY", "XGROUP", "XINFO"))
+# No key: what follows is a password, a message, a number or nothing.
+_REDIS_NO_KEY = frozenset(("PING", "ECHO", "INFO", "AUTH", "HELLO", "SELECT", "QUIT", "RESET", "MULTI", "EXEC", "DISCARD",
+                           "UNWATCH", "DBSIZE", "TIME", "FLUSHDB", "FLUSHALL", "SAVE", "BGSAVE", "BGREWRITEAOF", "LASTSAVE",
+                           "SCAN", "RANDOMKEY", "SWAPDB", "SHUTDOWN", "ROLE", "REPLICAOF", "SLAVEOF", "READONLY", "READWRITE",
+                           "LOLWUT", "FAILOVER", "PSYNC", "SYNC", "REPLCONF", "MIGRATE"))
+# Script calls: script or name, then how many keys, then the keys.
+_REDIS_SCRIPTS = frozenset(("EVAL", "EVALSHA", "EVAL_RO", "EVALSHA_RO", "FCALL", "FCALL_RO"))
+# How many keys first, then the keys.
+_REDIS_NUMKEYS = frozenset(("ZUNION", "ZINTER", "ZDIFF", "SINTERCARD", "ZINTERCARD", "LMPOP", "ZMPOP"))
+# Waiting, not working: a queue worker sits in these for seconds, so timing
+# them would put every idle worker at the top of the slow list.
+_REDIS_WAITING = frozenset(("BLPOP", "BRPOP", "BRPOPLPUSH", "BLMOVE", "BLMPOP", "BZPOPMIN", "BZPOPMAX", "BZMPOP", "WAIT",
+                            "WAITAOF", "SUBSCRIBE", "PSUBSCRIBE", "SSUBSCRIBE", "UNSUBSCRIBE", "PUNSUBSCRIBE",
+                            "SUNSUBSCRIBE", "MONITOR"))
+_REDIS_PREFIXES = frozenset(("PIPELINE", "MULTI"))
+# ASCII only, as JavaScript's /[a-z]/: str.upper() would change "ß" and "ı".
+_ASCII_UPPER = str.maketrans("abcdefghijklmnopqrstuvwxyz", "ABCDEFGHIJKLMNOPQRSTUVWXYZ")
+_JS_DECIMAL = re.compile(r"[+-]?(?:[0-9]+\.?[0-9]*|\.[0-9]+)(?:[eE][+-]?[0-9]+)?\Z")
+_JS_RADIX = re.compile(r"0[xXoObB][0-9a-fA-F]+\Z")
+_SPACE_RUN = re.compile(_S + "+")
+
+
+def _upper(text):
+    return text.translate(_ASCII_UPPER)
+
+
+def _positive(value):
+    """JavaScript's Number(value) > 0, for what a command's argument can be."""
+    if isinstance(value, bool):
+        return value
+    if isinstance(value, (int, float)):
+        return value > 0
+    if not isinstance(value, str):
+        return False
+    text = value.strip(_SPACE)
+    if _JS_DECIMAL.match(text):
+        return float(text) > 0
+    if _JS_RADIX.match(text):
+        try:
+            return int(text, 0) > 0
+        except ValueError:
+            return False
+    return text in ("Infinity", "+Infinity")
+
+
+def redis_key_shape(key):
+    # An empty key is "?" too: "GET " would come back from the collector as "GET".
+    if not isinstance(key, str) or key == "":
+        return "?"
+    return ":".join(part if part == "" or _REDIS_SEGMENT.match(part) else "?" for part in key.split(":"))
+
+
+def redis_command(args):
+    """A command as the client sends it (["GET", "user:42"]) -> "GET user:?",
+    or None for one that isn't timed (blocking, or not a command)."""
+    if not isinstance(args, (list, tuple)) or not args or not isinstance(args[0], str) or not _REDIS_WORD.match(args[0]):
+        return None
+    name = _upper(args[0])
+
+    def word(i):
+        return _upper(args[i]) if isinstance(args[i], str) else ""
+
+    if name in _REDIS_WAITING:
+        return None
+    command = name
+    key = None
+    if name in _REDIS_CONTAINERS:
+        if len(args) > 1 and isinstance(args[1], str) and _REDIS_WORD.match(args[1]):
+            command = name + " " + _upper(args[1])
+            # Only after a subcommand: "OBJECT user" would read back as OBJECT USER.
+            if name in _REDIS_SUB_KEY:
+                key = 2
+    elif name in _REDIS_SCRIPTS:
+        if len(args) > 2 and _positive(args[2]):
+            key = 3
+    elif name in _REDIS_NUMKEYS:
+        if len(args) > 1 and _positive(args[1]):
+            key = 2
+    elif name == "BITOP":
+        key = 2
+    elif name in ("XREAD", "XREADGROUP"):
+        # Its options, then STREAMS and the keys. XREADGROUP's group and
+        # consumer come first and may be named anything, as may a stream ("block").
+        at = -1
+        for i in range(4 if name == "XREADGROUP" else 1, len(args)):
+            if word(i) == "BLOCK":
+                return None
+            if word(i) == "STREAMS":
+                at = i
+                break
+        if at > 0:
+            key = at + 1
+    elif name not in _REDIS_NO_KEY and name not in _REDIS_PREFIXES:
+        key = 1
+    return command + " " + redis_key_shape(args[key]) if key is not None and key < len(args) else command
+
+
+def _redis_statement(text):
+    """A statement a package sent: [PIPELINE|MULTI] COMMAND [SUBCOMMAND] [key].
+    Anything else (an argument a package let through) is (other)."""
+    words = _SPACE_RUN.split(text.strip(_SPACE))
+    if words[0] == "":
+        return ""
+    out = []
+    i = 0
+    if _upper(words[0]) in _REDIS_PREFIXES and len(words) > 1:
+        out.append(_upper(words[0]))
+        i = 1
+    if not _REDIS_WORD.match(words[i]):
+        return OTHER_STATEMENT
+    name = _upper(words[i])
+    i += 1
+    if name in _REDIS_WAITING:
+        return OTHER_STATEMENT
+    out.append(name)
+    if name in _REDIS_CONTAINERS and i < len(words) and _REDIS_WORD.match(words[i]):
+        out.append(_upper(words[i]))
+        i += 1
+    if i < len(words) - 1:
+        return OTHER_STATEMENT
+    if i < len(words):
+        out.append(redis_key_shape(words[i]))
+    return " ".join(out)
+
+
 def normalize_statement(text, db):
     """The statement to store, or "" when nothing is left of it."""
     if not isinstance(text, str):
@@ -273,6 +416,8 @@ def normalize_statement(text, db):
     db = database_of(db)
     if db == "mongodb":
         shaped = _mongo_statement(text)
+    elif db == "redis":
+        shaped = _redis_statement(text)
     elif db == "other" and "\\" in text:
         # A database we don't know may or may not take a backslash as an
         # escape, and each reading leaks what the other hides. When they
@@ -1047,12 +1192,279 @@ def _clickhouse_client(connection, root):
     return None
 
 
+# ---- Redis clients: watch_database(client) (queries.md, "Redis") ----
+# Each command is one query under db "redis": its name and its first key's
+# shape (redis_command), never an argument. A pipeline or transaction is one,
+# timed as a whole: "PIPELINE " or "MULTI " and its first command. Blocking
+# commands (BLPOP, XREAD BLOCK) aren't timed. An error reply or no answer is
+# a failure; a nil reply isn't, and nor is a transaction a WATCH called off.
+#
+# The client itself is watched, not wrapped: its own execute_command (every
+# command goes through it) and pipeline() are replaced on that one object, so
+# it stays the client you made and nothing else is changed.
+
+_REDIS_MARK = "_analytics_platform_watched"
+# What redis-py splits a command's first argument on ("CLIENT SETNAME").
+_ASCII_SPACE = re.compile(r"[ \t\n\r\x0b\x0c]+")
+# Not what names a batch: its framing, and a WATCH a cluster pipeline sends at once.
+_REDIS_FRAMING = frozenset(("MULTI", "EXEC", "DISCARD", "WATCH", "UNWATCH"))
+
+
+@contextmanager
+def own_redis():
+    """The package reading Redis for itself: a watched client leaves it out."""
+    token = _own_redis.set(True)
+    try:
+        yield
+    finally:
+        _own_redis.reset(token)
+
+
+def _redis_arg(value):
+    """Bytes as text, as a key is read; a long value isn't a key worth
+    decoding. Anything else as it is (a number makes "?")."""
+    if isinstance(value, (bytes, bytearray, memoryview)):
+        return bytes(value).decode("utf-8", "replace") if len(value) <= 1024 else None
+    return value
+
+
+def _redis_list(args):
+    """The command as redis_command reads it: its first argument split into
+    words, as redis-py does to send it, then the rest. The key is at most the
+    third argument, so only the first few are taken, but all of XREAD's."""
+    if not args:
+        return None
+    first = _redis_arg(args[0])
+    head = [w for w in _ASCII_SPACE.split(first) if w] if isinstance(first, str) and _ASCII_SPACE.search(first) else [first]
+    name = head[0] if head else ""
+    end = len(args) if isinstance(name, str) and _upper(name) in ("XREAD", "XREADGROUP") else min(len(args), 5 - len(head))
+    return head + [_redis_arg(args[i]) for i in range(1, end)]
+
+
+def _redis_start(args):
+    """The statement a command starting now is timed as, or None."""
+    if _own_redis.get() or not _active():
+        return None
+    try:
+        return redis_command(_redis_list(args))
+    except Exception:
+        return None
+
+
+def _redis_failed(error):
+    # EXEC answering nil: the transaction a WATCH called off.
+    return error is not None and type(error).__name__ != "WatchError"
+
+
+def _replies_failed(replies):
+    """A pipeline run with raise_on_error=False hands its errors back."""
+    return isinstance(replies, (list, tuple)) and any(isinstance(r, Exception) and _redis_failed(r) for r in replies)
+
+
+def _redis_record(statement, started, error, replies=None):
+    try:
+        failed = _redis_failed(error) or _replies_failed(replies)
+        record("redis", statement, (time.perf_counter() - started) * 1000, failed)
+    except Exception:
+        pass  # never in place of the command's own result or error
+
+
+def _redis_timed(call, begin, batch=False):
+    """call (sync or async) timed as the statement begin(args) gives, or run
+    untimed when that's None. A batch's replies can hold its error replies."""
+    if inspect.iscoroutinefunction(call):
+        async def timed_async(*args, **kwargs):
+            statement = begin(args)
+            if statement is None:
+                return await call(*args, **kwargs)
+            started, error, result = time.perf_counter(), None, None
+            try:
+                result = await call(*args, **kwargs)
+                return result
+            except BaseException as raised:
+                error = raised
+                raise
+            finally:
+                _redis_record(statement, started, error, result if batch else None)
+        return timed_async
+
+    def timed(*args, **kwargs):
+        statement = begin(args)
+        if statement is None:
+            return call(*args, **kwargs)
+        started, error, result = time.perf_counter(), None, None
+        try:
+            result = call(*args, **kwargs)
+            return result
+        except BaseException as raised:
+            error = raised
+            raise
+        finally:
+            _redis_record(statement, started, error, result if batch else None)
+    return timed
+
+
+def _timed_batch(pipe, call, batch):
+    """call, a pipeline's execute, timed as one call named by its first command."""
+    def begin(_args):
+        first, batch[0] = batch[0], None
+        if first is None or _own_redis.get() or not _active():
+            return None
+        # transaction (sync), is_transaction (asyncio), _transaction (an asyncio
+        # cluster's); a sync cluster's says so only by the strategy it runs.
+        multi = any(getattr(pipe, name, False) is True
+                    for name in ("transaction", "is_transaction", "_transaction", "explicit_transaction")) \
+            or type(getattr(pipe, "_execution_strategy", None)).__name__ == "TransactionStrategy"
+        return ("MULTI " if multi else "PIPELINE ") + first
+
+    return _redis_timed(call, begin, batch=True)
+
+
+def _watch_redis_pipeline(pipe):
+    """A pipeline (or transaction) from a watched client: the commands queued
+    name it, execute() times it. The ones a WATCH makes run at once (redis-py's
+    immediate_execute_command) are timed on their own."""
+    batch = [None]
+
+    def note(args):
+        if batch[0] is None:
+            statement = _redis_start(args)
+            if statement and statement.split(" ", 1)[0] not in _REDIS_FRAMING:
+                batch[0] = statement
+
+    immediate = getattr(pipe, "immediate_execute_command", None)
+    staged = "pipeline_execute_command" if callable(immediate) and callable(getattr(pipe, "pipeline_execute_command", None)) \
+        else "execute_command"
+    queue = getattr(pipe, staged)
+
+    def queued(*args, **options):
+        note(args)
+        return queue(*args, **options)
+
+    setattr(pipe, staged, queued)
+    if staged == "pipeline_execute_command":
+        pipe.immediate_execute_command = _redis_timed(immediate, _redis_start)
+    reset = getattr(pipe, "reset", None)
+    if callable(reset):
+        def reset_batch(*args, **kwargs):
+            batch[0] = None
+            return reset(*args, **kwargs)
+        pipe.reset = reset_batch
+    pipe.execute = _timed_batch(pipe, pipe.execute, batch)
+    return pipe
+
+
+def _watch_redis(client):
+    """A redis-py client (sync or asyncio, Redis or RedisCluster), in place."""
+    if getattr(client, _REDIS_MARK, False):
+        return client
+    client.execute_command = _redis_timed(client.execute_command, _redis_start)
+    pipeline = getattr(client, "pipeline", None)
+    if callable(pipeline):
+        def watched_pipeline(*args, **kwargs):
+            pipe = pipeline(*args, **kwargs)
+            try:
+                _watch_redis_pipeline(pipe)
+            except Exception:
+                pass  # its commands still run, untimed
+            return pipe
+        client.pipeline = watched_pipeline
+    setattr(client, _REDIS_MARK, True)
+    return client
+
+
+def _is_redis_client(client):
+    """redis-py's (or valkey-py's) Redis or RedisCluster, or a subclass of one."""
+    if not callable(getattr(client, "execute_command", None)) or not callable(getattr(client, "pipeline", None)):
+        return False
+    return any(klass.__module__.split(".")[0] in ("redis", "valkey") for klass in type(client).__mro__)
+
+
+def _making_watched(make):
+    """make, a function that makes (or connects) a Redis client, made to
+    hand back each client watched."""
+    def make_watched(*args, **kwargs):
+        client = make(*args, **kwargs)
+        # Inside the app's cache call: a client we can't watch (a custom
+        # CLIENT_CLASS's) is handed back as it is, never an error.
+        try:
+            if _is_redis_client(client):
+                _watch_redis(client)
+        except Exception:
+            pass
+        return client
+    setattr(make_watched, _REDIS_MARK, True)
+    return make_watched
+
+
+def _watch_django_cache(cache):
+    """A Django cache on Redis: Django's own RedisCache makes a client for each
+    operation, so the clients it makes are watched as it makes them;
+    django-redis keeps its clients, and those it has and connects are."""
+    module = type(cache).__module__
+    if module.startswith("django.core.cache.backends.redis"):
+        inner = cache._cache
+        if not getattr(inner._client, _REDIS_MARK, False):
+            inner._client = _making_watched(inner._client)
+        return True
+    if module.startswith("django_redis"):
+        inner = cache.client
+        connect = getattr(inner, "connect", None)
+        if not callable(connect) or getattr(connect, _REDIS_MARK, False):
+            return callable(connect)
+        inner.connect = _making_watched(connect)
+        held = list(getattr(inner, "_clients", None) or []) + list((getattr(inner, "_serverdict", None) or {}).values())
+        for client in held:
+            if client is not None and _is_redis_client(client):
+                _watch_redis(client)
+        return True
+    return False
+
+
+def _watch_django_alias(cache):
+    """Django keeps a cache object per thread: the one watched is this
+    thread's. The others of its alias are watched as Django makes them."""
+    try:
+        from django.core.cache import caches
+    except Exception:
+        return
+    try:
+        alias = next((name for name in caches if caches[name] is cache), None)
+        if alias is None:
+            return
+        aliases = getattr(caches, _REDIS_MARK, None)
+        if aliases is None:
+            create = caches.create_connection
+
+            def create_watched(name, *args, **kwargs):
+                made = create(name, *args, **kwargs)
+                if name in aliases:
+                    try:
+                        _watch_django_cache(made)
+                    except Exception:
+                        pass  # that thread's cache still works, untimed
+                return made
+            aliases = set()
+            caches.create_connection = create_watched
+            setattr(caches, _REDIS_MARK, aliases)
+        aliases.add(alias)
+    except Exception:
+        pass  # this thread's cache is still watched
+
+
 def watch_database(connection, db=None):
-    """A DB-API connection, or a clickhouse-connect / clickhouse-driver
-    client, whose queries are timed: use what it returns in place of the
-    connection. db: one of DATABASES (default: by its driver)."""
+    """A DB-API connection, a clickhouse-connect / clickhouse-driver client,
+    a redis-py client or a Django cache on Redis, whose queries are timed:
+    use what it returns in place of the connection (a Redis client or cache
+    comes back as itself, watched). db: one of DATABASES (default: by its driver)."""
     if isinstance(connection, (_Connection, _ClickHouse)):
         return connection
+    if _is_redis_client(connection):
+        return _watch_redis(connection)
+    if type(connection).__module__.startswith(("django.core.cache.backends.redis", "django_redis")):
+        if _watch_django_cache(connection):
+            _watch_django_alias(connection)
+        return connection
     root = type(connection).__module__.split(".")[0]
     client = _clickhouse_client(connection, root)
     if client is not None:

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform/queues.py

@@ -23,6 +23,7 @@ import weakref
 from datetime import datetime, timezone
 
 from .core import Periodic
+from .queries import own_redis
 
 SAMPLE_EVERY = 60
 MAX_QUEUES = 100
@@ -126,7 +127,10 @@ class Queues:
         for entry in list(self.entries.values()):
             gauges = None
             try:
-                gauges = entry.read()
+                # Through the app's own connection (RQ's), maybe watched:
+                # our reads aren't its commands.
+                with own_redis():
+                    gauges = entry.read()
             except Exception as error:
                 self.core.log("performance", "could not read queue %s:" % entry.queue, error)
             with self.lock:

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: analytics-platform
-Version: 0.3.5
+Version: 0.3.6
 Summary: Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard.
 Author: steviec87
 License-Expression: MIT
@@ -317,6 +317,8 @@ read, so no data leaves it.
 | pymongo | nothing, but **create the platform before your `MongoClient`**: pymongo only reports to clients made after the package is listening |
 | Any other DB-API driver (psycopg, psycopg2, mysqlclient, PyMySQL, sqlite3...) | `conn = platform.watch_database(psycopg.connect(...))`, and use `conn` as before |
 | ClickHouse: clickhouse-connect, clickhouse-driver | `client = platform.watch_database(clickhouse_connect.get_client(...))` (or a `clickhouse_driver.Client`), and use `client` as before |
+| Redis: redis-py (`Redis`, `RedisCluster`, sync or `redis.asyncio`) | `platform.watch_database(r)`: the client is watched in place, so keep using `r` |
+| Django's cache on Redis (Django's own `RedisCache`, or django-redis) | `platform.watch_database(caches["default"])`, once at start-up (in `AppConfig.ready()`) |
 
 A query outside any request or job is counted as `JOB (background)`. Each
 query is counted once: when SQLAlchemy or Django runs it over a watched
@@ -331,6 +333,23 @@ they answer. clickhouse-driver's `execute()` likewise. The streaming
 queries (`query_rows_stream()`, `execute_iter()`) aren't timed. Only the
 client you pass is wrapped; the library itself is left alone.
 
+Redis: each command is counted by its name and the shape of its key (`GET
+user:?:cart`); no other argument or value is read. A pipeline or transaction
+is one call (`PIPELINE GET user:?`, `MULTI SET cart:?`), timed as a whole.
+Commands that wait (`BLPOP`, `XREAD BLOCK`, subscribing) aren't timed, and
+nor are this package's own reads of your RQ queues. An error reply counts
+as failed; a nil reply doesn't.
+
+```python
+r = redis.Redis.from_url(REDIS_URL)
+platform.watch_database(r)
+
+# Django (settings: CACHES backend django.core.cache.backends.redis.RedisCache
+# or django_redis.cache.RedisCache), in your AppConfig.ready():
+from django.core.cache import caches
+platform.watch_database(caches["default"])
+```
+
 ## Outgoing calls
 
 Health → Server speed → Calls shows how long each route waited on other

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/analytics_platform.egg-info/SOURCES.txt

@@ -41,6 +41,7 @@ tests/test_limits.py
 tests/test_page_limits.py
 tests/test_proxy.py
 tests/test_queries.py
+tests/test_redis.py
 tests/test_request_rules.py
 tests/test_resources.py
 tests/test_rq_dramatiq.py

{analytics_platform-0.3.5 → analytics_platform-0.3.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "analytics-platform"
-version = "0.3.5"
+version = "0.3.6"
 description = "Firewall, error monitoring and server analytics for your Python server, run from your analytics dashboard."
 readme = "README.md"
 requires-python = ">=3.8"

analytics_platform-0.3.6/tests/test_redis.py

@@ -0,0 +1,428 @@
+"""Redis command timings (sdk/protocols/queries.md, "Redis"): redis_command
+against the shared cases, and the redis-py hooks (sync, asyncio, pipelines,
+transactions, a Django cache) against stand-ins shaped as redis-py's objects
+are; the package doesn't depend on redis-py.
+
+    python3 -m unittest discover -s tests
+"""
+import asyncio
+import json
+import os
+import sys
+import threading
+import types
+import unittest
+from unittest import mock
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from analytics_platform import create_platform  # noqa: E402
+from analytics_platform import queries as q  # noqa: E402
+
+CASES = os.path.join(os.path.dirname(__file__), "..", "..", "protocols", "query-cases.json")
+SITE = "a1b2c3d4e5f6a1b2c3d4e5f6"
+
+
+def make_platform(**options):
+    return create_platform(site_id=SITE, secret="s", endpoint="http://127.0.0.1:9",
+                           firewall=False, analytics=False, flags=False, inventory=False, posture=False,
+                           resources=False, proxy=False, maintenance=False, crawlers=False, forms=False,
+                           audit=False, headers=False, errors=False, **options)
+
+
+class ResponseError(Exception):
+    pass
+
+
+class WatchError(Exception):
+    pass
+
+
+def reply(args):
+    """What the stand-in server answers: nil for "missing", an error reply for "text"."""
+    key = args[1] if len(args) > 1 else None
+    if isinstance(key, bytes):
+        key = key.decode()
+    if key == "missing":
+        return None
+    if key == "text":
+        raise ResponseError("WRONGTYPE Operation against a key holding the wrong kind of value")
+    return b"v"
+
+
+# ---- redis-py's shapes: every command through the client's execute_command;
+# a Pipeline (a Redis subclass) stages commands, runs WATCH and what follows
+# it at once, and sends the rest on execute() ----
+
+class Redis:
+    def __init__(self):
+        self.sent = []
+
+    def execute_command(self, *args, **options):
+        self.sent.append(args)
+        return reply(args)
+
+    def get(self, name):
+        return self.execute_command("GET", name)
+
+    def incr(self, name):
+        return self.execute_command("INCR", name)
+
+    def client_setname(self, name):
+        return self.execute_command("CLIENT SETNAME", name)
+
+    def blpop(self, keys, timeout=0):
+        return self.execute_command("BLPOP", *keys, timeout)
+
+    def eval(self, script, numkeys, *keys_and_args):
+        return self.execute_command("EVAL", script, numkeys, *keys_and_args)
+
+    def pipeline(self, transaction=True, shard_hint=None):
+        return Pipeline(self, transaction)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc):
+        return None
+
+
+class Pipeline(Redis):
+    def __init__(self, redis, transaction):
+        super().__init__()
+        self.redis = redis
+        self.transaction = transaction
+        self.stack = []
+        self.watching = False
+        self.explicit_transaction = False
+
+    def execute_command(self, *args, **kwargs):
+        if (self.watching or args[0] == "WATCH") and not self.explicit_transaction:
+            return self.immediate_execute_command(*args, **kwargs)
+        return self.pipeline_execute_command(*args, **kwargs)
+
+    def immediate_execute_command(self, *args, **options):
+        if args[0] == "WATCH":
+            self.watching = True
+            return True
+        return reply(args)
+
+    def pipeline_execute_command(self, *args, **options):
+        self.stack.append(args)
+        return self
+
+    def watch(self, *names):
+        return self.execute_command("WATCH", *names)
+
+    def multi(self):
+        self.explicit_transaction = True
+
+    def reset(self):
+        self.stack, self.watching, self.explicit_transaction = [], False, False
+
+    def execute(self, raise_on_error=True):
+        try:
+            if any(args[1:2] == ("changed",) for args in self.stack):
+                raise WatchError("Watched variable changed.")
+            replies = []
+            for args in self.stack:
+                try:
+                    replies.append(reply(args))
+                except ResponseError as error:
+                    if raise_on_error:
+                        raise
+                    replies.append(error)
+            return replies
+        finally:
+            Pipeline.reset(self)
+
+
+class AsyncRedis(Redis):
+    async def execute_command(self, *args, **options):
+        await asyncio.sleep(0)
+        return reply(args)
+
+    def pipeline(self, transaction=True, shard_hint=None):
+        return AsyncPipeline(self, transaction)
+
+
+class AsyncPipeline(Pipeline):
+    async def immediate_execute_command(self, *args, **options):
+        return Pipeline.immediate_execute_command(self, *args, **options)
+
+    async def execute(self, raise_on_error=True):
+        await asyncio.sleep(0)
+        return Pipeline.execute(self, raise_on_error)
+
+    async def reset(self):
+        Pipeline.reset(self)
+
+
+# Detected by their module, as redis-py's own are.
+for _class in (Redis, Pipeline, AsyncRedis, AsyncPipeline):
+    _class.__module__ = "redis.client"
+
+
+class SharedCases(unittest.TestCase):
+    def test_every_shared_redis_command_case_comes_out_character_for_character(self):
+        with open(CASES, encoding="utf-8") as handle:
+            cases = json.load(handle)
+        self.assertGreater(len(cases["redis_commands"]), 20)
+        for case in cases["redis_commands"]:
+            self.assertEqual(q.redis_command(case["args"]), case["expected"], repr(case["args"]))
+
+    def test_unicode_letters_are_not_ascii_letters(self):
+        self.assertIsNone(q.redis_command(["ſet", "k"]))
+        self.assertEqual(q.redis_command(["SET", "straße:1"]), "SET ?:?")
+        self.assertEqual(q.normalize_statement("xread　k", "redis"), "XREAD k")
+
+
+class Hooks(unittest.TestCase):
+    def setUp(self):
+        self.platform = make_platform()
+        self.queries = self.platform._modules["performance"].queries
+        self.queries.rows.clear()
+        self.queries.per_minute.clear()
+
+    def tearDown(self):
+        self.queries.rows.clear()
+        self.queries.retry = []
+        self.platform.close()
+
+    def redis_rows(self):
+        return {key[4]: (row["count"], row["errors"]) for key, row in self.queries.rows.items() if key[3] == "redis"}
+
+    def test_each_command_is_timed_with_errors_counted_nil_not_and_blocking_ones_left_out(self):
+        original = Redis.__dict__["execute_command"]
+        redis = Redis()
+        self.assertIs(self.platform.watch_database(redis), redis)
+        self.assertIs(self.platform.watch_database(redis), redis)
+        self.assertEqual(redis.get("user:42:cart"), b"v")
+        self.assertEqual(redis.get(b"user:7:cart"), b"v")
+        self.assertIsNone(redis.get("missing"))
+        with self.assertRaises(ResponseError):
+            redis.incr("text")
+        redis.client_setname("web-1")
+        redis.eval("return 1", 1, "lock:order:9", "hunter2")
+        redis.blpop(["jobs"], 0)
+        self.assertEqual(self.redis_rows(), {
+            "GET user:?:cart": (2, 0), "GET missing": (1, 0), "INCR text": (1, 1),
+            "CLIENT SETNAME": (1, 0), "EVAL lock:order:?": (1, 0)})
+        self.assertIs(Redis.__dict__["execute_command"], original, "the class is left alone")
+
+    def test_a_pipeline_or_transaction_is_one_call_named_by_its_first_command(self):
+        redis = self.platform.watch_database(Redis())
+        pipe = redis.pipeline(transaction=False)
+        pipe.get("user:1")
+        pipe.incr("user:2")
+        self.assertEqual(pipe.execute(), [b"v", b"v"])
+        with redis.pipeline() as tx:
+            tx.execute_command("SET", "cart:9", "x")
+            tx.get("cart:9")
+            tx.execute()
+        broken = redis.pipeline(transaction=False)
+        broken.get("user:3")
+        broken.incr("text")
+        self.assertIsInstance(broken.execute(raise_on_error=False)[1], ResponseError)
+        # An empty one isn't a call.
+        redis.pipeline().execute()
+        self.assertEqual(self.redis_rows(), {"PIPELINE GET user:?": (2, 1), "MULTI SET cart:?": (1, 0)})
+
+    def test_a_watch_runs_its_commands_at_once_and_a_called_off_transaction_isnt_a_failure(self):
+        redis = self.platform.watch_database(Redis())
+        pipe = redis.pipeline()
+        pipe.watch("stock:9")
+        pipe.get("stock:9")
+        pipe.multi()
+        pipe.execute_command("SET", "changed", "1")
+        with self.assertRaises(WatchError):
+            pipe.execute()
+        self.assertEqual(self.redis_rows(), {"WATCH stock:?": (1, 0), "GET stock:?": (1, 0), "MULTI SET changed": (1, 0)})
+
+    def test_asyncio_clients_and_pipelines(self):
+        redis = self.platform.watch_database(AsyncRedis())
+
+        async def run():
+            self.assertEqual(await redis.get("user:42"), b"v")
+            with self.assertRaises(ResponseError):
+                await redis.incr("text")
+            pipe = redis.pipeline(transaction=False)
+            pipe.get("user:1")
+            pipe.get("user:2")
+            self.assertEqual(await pipe.execute(), [b"v", b"v"])
+
+        asyncio.run(run())
+        self.assertEqual(self.redis_rows(), {"GET user:?": (1, 0), "INCR text": (1, 1), "PIPELINE GET user:?": (1, 0)})
+
+    def test_the_packages_own_queue_reads_on_the_same_client_arent_timed(self):
+        redis = self.platform.watch_database(Redis())
+        queues = self.platform._modules["performance"].queues
+
+        def read():
+            pipe = redis.pipeline(transaction=False)
+            pipe.execute_command("LLEN", "rq:queue:default")
+            pipe.execute()
+            redis.execute_command("HGET", "rq:job:1", "enqueued_at")
+            return {"waiting": 0}
+
+        queues.entry("rq", "localhost:6379/0", "default", read)
+        with mock.patch.object(queues.core, "send"):
+            queues.sample()
+        redis.get("user:1")
+        self.assertEqual(self.redis_rows(), {"GET user:?": (1, 0)})
+
+    def test_djangos_redis_cache_watches_each_client_it_makes(self):
+        class RedisCacheClient:
+            def __init__(self):
+                self._client = Redis
+
+            def get_client(self, key=None, *, write=False):
+                return self._client()
+
+        class RedisCache:
+            def __init__(self):
+                self._cache = RedisCacheClient()
+
+            def get(self, key):
+                return self._cache.get_client(key).get(":1:" + key)
+
+        RedisCache.__module__ = "django.core.cache.backends.redis"
+        cache = RedisCache()
+        self.assertIs(self.platform.watch_database(cache), cache)
+        self.platform.watch_database(cache)
+        cache.get("views")
+        cache.get("views")
+        self.assertEqual(self.redis_rows(), {"GET :?:views": (2, 0)})
+
+    def test_a_django_cache_made_for_another_thread_is_watched_too(self):
+        class RedisCacheClient:
+            def get_client(self, key=None, *, write=False):
+                return self._client()
+
+        class RedisCache:
+            def __init__(self):
+                self._cache = RedisCacheClient()
+                self._cache._client = Redis
+
+            def get(self, key):
+                return self._cache.get_client(key).get(":1:" + key)
+
+        RedisCache.__module__ = "django.core.cache.backends.redis"
+
+        class CacheHandler:
+            """As Django's caches: one cache object per alias per thread."""
+            def __init__(self):
+                self.local = threading.local()
+
+            def __iter__(self):
+                return iter(["default", "other"])
+
+            def create_connection(self, alias):
+                return RedisCache()
+
+            def __getitem__(self, alias):
+                made = getattr(self.local, alias, None)
+                if made is None:
+                    made = self.create_connection(alias)
+                    setattr(self.local, alias, made)
+                return made
+
+        caches = CacheHandler()
+        module = types.ModuleType("django.core.cache")
+        module.caches = caches
+        with mock.patch.dict(sys.modules, {"django": types.ModuleType("django"), "django.core": types.ModuleType("django.core"),
+                                           "django.core.cache": module}):
+            self.platform.watch_database(caches["default"])
+            self.platform.watch_database(caches["default"])
+            caches["default"].get("views")
+            thread = threading.Thread(target=lambda: (caches["default"].get("views"), caches["other"].get("views")))
+            thread.start()
+            thread.join()
+        self.assertEqual(self.redis_rows(), {"GET :?:views": (2, 0)})
+
+    def test_a_sync_cluster_transaction_is_multi_and_its_watch_doesnt_name_it(self):
+        class TransactionStrategy:
+            pass
+
+        class ClusterPipeline(Redis):
+            def __init__(self):
+                super().__init__()
+                self._execution_strategy = TransactionStrategy()
+                self.stack = []
+
+            def execute_command(self, *args, **kwargs):
+                if args[0] != "WATCH":
+                    self.stack.append(args)
+                return self
+
+            def execute(self, raise_on_error=True):
+                stack, self.stack = self.stack, []
+                return [reply(args) for args in stack]
+
+        class RedisCluster(Redis):
+            def pipeline(self, transaction=None, shard_hint=None):
+                return ClusterPipeline()
+
+        RedisCluster.__module__ = ClusterPipeline.__module__ = "redis.cluster"
+        redis = self.platform.watch_database(RedisCluster())
+        pipe = redis.pipeline(transaction=True)
+        pipe.execute_command("WATCH", "stock:9")
+        pipe.get("stock:9")
+        self.assertEqual(pipe.execute(), [b"v"])
+        self.assertEqual(self.redis_rows(), {"MULTI GET stock:?": (1, 0)})
+
+    def test_django_redis_watches_the_clients_it_holds_and_connects(self):
+        held = Redis()
+
+        class DefaultClient:
+            def __init__(self):
+                self._clients = [held, None]
+
+            def connect(self, index=0):
+                return Redis()
+
+            def get_client(self, write=True):
+                if self._clients[1] is None:
+                    self._clients[1] = self.connect(1)
+                return self._clients[0 if write else 1]
+
+        class RedisCache:
+            client = DefaultClient()
+
+        RedisCache.__module__ = "django_redis.cache"
+        cache = RedisCache()
+        self.platform.watch_database(cache)
+        cache.client.get_client(write=True).get("session:a1")
+        cache.client.get_client(write=False).get("session:b2")
+        self.assertEqual(self.redis_rows(), {"GET session:?": (2, 0)})
+
+    def test_a_django_redis_client_that_isnt_redis_pys_is_handed_back_untouched(self):
+        class Custom:
+            """A CLIENT_CLASS's connection: no execute_command to watch."""
+            __slots__ = ()
+
+        class DefaultClient:
+            _clients = [None]
+
+            def connect(self, index=0):
+                return Custom()
+
+        class RedisCache:
+            client = DefaultClient()
+
+        RedisCache.__module__ = "django_redis.cache"
+        cache = RedisCache()
+        self.platform.watch_database(cache)
+        self.assertIsInstance(cache.client.connect(0), Custom)
+
+    def test_with_server_speed_off_nothing_is_timed(self):
+        platform = make_platform(performance=False)
+        redis = Redis()
+        self.assertIs(platform.watch_database(redis), redis)
+        redis.get("user:1")
+        platform.close()
+        self.assertEqual(self.redis_rows(), {})
+
+
+if __name__ == "__main__":
+    unittest.main()