kedis-python 0.1.4__tar.gz → 0.2.0__tar.gz

{kedis_python-0.1.4 → kedis_python-0.2.0}/PKG-INFO

@@ -1,10 +1,16 @@
 Metadata-Version: 2.4
 Name: kedis-python
-Version: 0.1.4
+Version: 0.2.0
 Summary: A Redis-inspired in-memory store in Python.
 Author: V SS Karthik
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/Hogwarts-coder10/kedis-python
+Keywords: redis,in-memory,key-value,database,streams,pub-sub
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Database
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -43,16 +49,17 @@ Rather than focusing on Redis compatibility, Kedis focuses on understanding the
 ## 📚 Engine Documentation
 
 Dive into the internal grimoire to understand how Kedis achieves microsecond latency:
-* [Architecture & Pipeline](docs/architecture.md)
-* [KESP Wire Protocol](docs/protocol.md)
-* [Networking & Surge Tank](docs/networking.md)
-* [Persistence & I/O Drivetrain](docs/persistence.md)
-* [Command Spellbook](docs/commands.md)
-* [TTL & Lifespan Mechanics](docs/ttl.md)
-* [Streams & Consumer Groups](docs/streams.md)
-* [Pub/Sub Broadcasts](docs/pubsub.md)
-* [Benchmarks & Telemetry](docs/benchmarks.md)
-* [Future Roadmap](docs/roadmap.md)
+* [Architecture & Pipeline](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/architecture.md)
+* [KESP Wire Protocol](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/protocol.md)
+* [Networking & Surge Tank](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/networking.md)
+* [Limits & Budgets](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/limits.md)
+* [Persistence & I/O Drivetrain](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/persistence.md)
+* [Command Spellbook](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/commands.md)
+* [TTL & Lifespan Mechanics](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/ttl.md)
+* [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md)
+* [Pub/Sub Broadcasts](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/pubsub.md)
+* [Benchmarks & Telemetry](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/benchmarks.md)
+* [Future Roadmap](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/roadmap.md)
 
 ---
 
@@ -120,7 +127,7 @@ XPENDING orders shipping
 XACK orders shipping 1718000000000-0
 ```
 
-Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](docs/streams.md).
+Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md).
 
 ---
 
@@ -377,7 +384,7 @@ Current limitations include:
 * limited fault tolerance compared to production databases
 * no memory-based eviction ceiling (LRU tracks usage, but isn't yet tied to a hard memory limit)
 * AOF persistence is not buffered — writes can trigger a sync depending on config
-* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](docs/streams.md#-known-limits)
+* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md#-known-limits)
 
 These limitations are intentional learning opportunities and areas of active development.
 

{kedis_python-0.1.4 → kedis_python-0.2.0}/README.md

@@ -29,16 +29,17 @@ Rather than focusing on Redis compatibility, Kedis focuses on understanding the
 ## 📚 Engine Documentation
 
 Dive into the internal grimoire to understand how Kedis achieves microsecond latency:
-* [Architecture & Pipeline](docs/architecture.md)
-* [KESP Wire Protocol](docs/protocol.md)
-* [Networking & Surge Tank](docs/networking.md)
-* [Persistence & I/O Drivetrain](docs/persistence.md)
-* [Command Spellbook](docs/commands.md)
-* [TTL & Lifespan Mechanics](docs/ttl.md)
-* [Streams & Consumer Groups](docs/streams.md)
-* [Pub/Sub Broadcasts](docs/pubsub.md)
-* [Benchmarks & Telemetry](docs/benchmarks.md)
-* [Future Roadmap](docs/roadmap.md)
+* [Architecture & Pipeline](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/architecture.md)
+* [KESP Wire Protocol](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/protocol.md)
+* [Networking & Surge Tank](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/networking.md)
+* [Limits & Budgets](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/limits.md)
+* [Persistence & I/O Drivetrain](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/persistence.md)
+* [Command Spellbook](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/commands.md)
+* [TTL & Lifespan Mechanics](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/ttl.md)
+* [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md)
+* [Pub/Sub Broadcasts](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/pubsub.md)
+* [Benchmarks & Telemetry](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/benchmarks.md)
+* [Future Roadmap](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/roadmap.md)
 
 ---
 
@@ -106,7 +107,7 @@ XPENDING orders shipping
 XACK orders shipping 1718000000000-0
 ```
 
-Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](docs/streams.md).
+Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md).
 
 ---
 
@@ -363,7 +364,7 @@ Current limitations include:
 * limited fault tolerance compared to production databases
 * no memory-based eviction ceiling (LRU tracks usage, but isn't yet tied to a hard memory limit)
 * AOF persistence is not buffered — writes can trigger a sync depending on config
-* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](docs/streams.md#-known-limits)
+* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md#-known-limits)
 
 These limitations are intentional learning opportunities and areas of active development.
 

kedis_python-0.2.0/kedis_python/aof.py

@@ -0,0 +1,138 @@
+"""On-disk record format for the append-only file (AOF).
+
+Every logged command is one KESP-framed record, the same shape the wire
+protocol uses for commands:
+
+    A<count>\\n  then, for each token:  S<length>\\n<payload>\\n
+
+Lengths count bytes of the UTF-8 payload, so a token can hold anything: spaces,
+newlines, tabs, an empty string, text that looks like another command. The old
+format joined tokens with spaces on one line, so a value could not survive a
+restart and a value containing a newline could plant commands in the log.
+
+Strings are encoded with "surrogatepass", so every Python str round-trips
+exactly, lone surrogates included.
+
+Files written by older versions hold one whitespace-separated command per line.
+scan() still reads those, line by line, and a file may mix both: old lines
+first, framed records after.
+"""
+
+import re
+
+_COUNT = re.compile(rb"(?:0|[1-9][0-9]*)")
+MAX_HEADER_LINE = 32  # "A<n>" / "S<n>" lines are never longer than this
+
+
+def encode_record(tokens) -> bytes:
+    """One command as a framed record."""
+    parts = [b"A%d\n" % len(tokens)]
+    for token in tokens:
+        raw = str(token).encode("utf-8", "surrogatepass")
+        parts.append(b"S%d\n" % len(raw))
+        parts.append(raw)
+        parts.append(b"\n")
+    return b"".join(parts)
+
+
+class ScanResult:
+    """How a scan ended.
+
+    status:   "ok"      every byte was consumed
+              "torn"    the file ends in the middle of a record (a crash while
+                        writing); everything before good_end is intact
+              "corrupt" a record is malformed; see reason
+    good_end: offset just past the last complete record or line
+    """
+
+    __slots__ = ("good_end", "reason", "status")
+
+    def __init__(self):
+        self.status = "ok"
+        self.good_end = 0
+        self.reason = ""
+
+
+def _count(buf, start: int, end: int):
+    field = bytes(buf[start:end])
+    return int(field) if _COUNT.fullmatch(field) else None
+
+
+def scan(buf, result: ScanResult):
+    """Yields (tokens, end_offset) for each command in buf, a bytes-like object
+    that supports find() and slicing (bytes or mmap). Stops at the first
+    incomplete or malformed record and says why in result."""
+    total = len(buf)
+    pos = 0
+
+    def stop(status: str, reason: str):
+        result.status = status
+        result.reason = reason
+
+    while pos < total:
+        if buf[pos : pos + 1] == b"A":
+            following = buf[pos + 1 : pos + 2]
+            if following == b"":
+                return stop("torn", "file ends inside a record header")
+            # A legacy command word starts "A" + a letter (AUTH, ...). Anything
+            # else after the "A" is a record header and must validate as one.
+            framed = not following.isalpha()
+        else:
+            framed = False
+
+        if not framed:
+            # Legacy line: whitespace-separated tokens, as older versions wrote.
+            nl = buf.find(b"\n", pos)
+            line_end = total if nl == -1 else nl
+            next_pos = total if nl == -1 else nl + 1
+            tokens = bytes(buf[pos:line_end]).decode("utf-8", "replace").split()
+            result.good_end = next_pos
+            if tokens:
+                yield tokens, next_pos
+            pos = next_pos
+            continue
+
+        nl = buf.find(b"\n", pos)
+        if nl == -1:
+            if total - pos > MAX_HEADER_LINE:
+                return stop("corrupt", "array header too long")
+            return stop("torn", "file ends inside a record header")
+        if nl - pos > MAX_HEADER_LINE:
+            return stop("corrupt", "array header too long")
+        count = _count(buf, pos + 1, nl)
+        if count is None:
+            return stop("corrupt", "invalid array header")
+
+        cursor = nl + 1
+        tokens = []
+        for _ in range(count):
+            if cursor >= total:
+                return stop("torn", "file ends between tokens")
+            if buf[cursor : cursor + 1] != b"S":
+                return stop("corrupt", "expected a string header")
+            nl = buf.find(b"\n", cursor)
+            if nl == -1:
+                if total - cursor > MAX_HEADER_LINE:
+                    return stop("corrupt", "string header too long")
+                return stop("torn", "file ends inside a string header")
+            if nl - cursor > MAX_HEADER_LINE:
+                return stop("corrupt", "string header too long")
+            length = _count(buf, cursor + 1, nl)
+            if length is None:
+                return stop("corrupt", "invalid string length")
+            start = nl + 1
+            end = start + length
+            if end + 1 > total:
+                return stop("torn", "file ends inside a string")
+            if buf[end : end + 1] != b"\n":
+                return stop("corrupt", "string is not newline-terminated")
+            try:
+                tokens.append(bytes(buf[start:end]).decode("utf-8", "surrogatepass"))
+            except UnicodeDecodeError:
+                return stop("corrupt", "string is not valid UTF-8")
+            cursor = end + 1
+
+        result.good_end = cursor
+        if tokens:
+            yield tokens, cursor
+        pos = cursor

{kedis_python-0.1.4 → kedis_python-0.2.0}/kedis_python/commands.py

@@ -1,13 +1,17 @@
+import math
 import threading
 import time
 from typing import Any
 
-from .parser import BulkString
+from .parser import BulkString, KESPEncoder
 from .store import KedisStore
 from .stream import StreamError
 
 
 class CommandHandler:
+    # Bytes a pub/sub subscriber may leave unread before the server cuts it off.
+    MAX_SUBSCRIBER_BUFFER = 32 * 1024 * 1024
+
     def __init__(self, store: KedisStore):
         self.store = store
 
@@ -16,6 +20,9 @@ class CommandHandler:
 
         # Pub / Sub SwitchBoard
         self._channels = {}
+        # The server's event loop, set per connection; pub/sub pushes are queued
+        # onto it because commands run in worker threads.
+        self.event_loop = None
 
         # The O(1) Dispatch Table
         self._commands = {
@@ -215,23 +222,30 @@ class CommandHandler:
         self.store.set(tokens[1], tokens[2])
         return "OK"
 
+    @staticmethod
+    def _data(value):
+        """A stored value as a reply: always data, never read as a status or
+        an error, whatever it looks like ("OK", "ERROR x", "-ERR y")."""
+        return BulkString(value) if isinstance(value, str) else value
+
     def _handle_get(self, tokens: list[str]):
         if len(tokens) != 2:
             return "-ERR wrong number of arguments for 'GET' command"
         try:
-            return self.store.get(tokens[1])
+            return self._data(self.store.get(tokens[1]))
         except TypeError as e:
             return f"-ERR {str(e)}"
 
     def _handle_del(self, tokens: list[str]):
-        if len(tokens) != 2:
+        if len(tokens) < 2:
             return "-ERR wrong number of arguments for 'DEL' command"
-        return self.store.delete(tokens[1])
+        return sum(self.store.delete(key) for key in tokens[1:])
 
     def _handle_exists(self, tokens: list[str]):
-        if len(tokens) != 2:
+        if len(tokens) < 2:
             return "-ERR wrong number of arguments for 'EXISTS' command"
-        return self.store.exists(tokens[1])
+        # Like Redis, a key named twice counts twice.
+        return sum(self.store.exists(key) for key in tokens[1:])
 
     def _handle_expire(self, tokens: list[str]):
         if len(tokens) != 3:
@@ -310,7 +324,7 @@ class CommandHandler:
         if len(tokens) != 2:
             return "-ERR wrong number of arguments for 'lpop' command"
         try:
-            return self.store.lpop(tokens[1])
+            return self._data(self.store.lpop(tokens[1]))
         except TypeError as e:
             return f"-ERR {str(e)}"
 
@@ -318,7 +332,7 @@ class CommandHandler:
         if len(tokens) != 2:
             return "-ERR wrong number of arguments for 'rpop' command"
         try:
-            return self.store.rpop(tokens[1])
+            return self._data(self.store.rpop(tokens[1]))
         except TypeError as e:
             return f"-ERR {str(e)}"
 
@@ -360,7 +374,7 @@ class CommandHandler:
         if len(tokens) != 3:
             return "-ERR wrong number of arguments for 'hget' command"
         try:
-            return self.store.hget(tokens[1], tokens[2])
+            return self._data(self.store.hget(tokens[1], tokens[2]))
         except TypeError as e:
             return f"-ERR {str(e)}"
 
@@ -386,9 +400,15 @@ class CommandHandler:
         key = tokens[1]
         added = 0
         try:
+            # Validate every pair before applying any: a bad score anywhere
+            # means nothing is added.
+            pairs = []
             for i in range(2, len(tokens), 2):
                 score = float(tokens[i])
-                member = tokens[i + 1]
+                if math.isnan(score):
+                    raise ValueError
+                pairs.append((score, tokens[i + 1]))
+            for score, member in pairs:
                 added += self.store.zadd(key, score, member)
             return added
         except ValueError:
@@ -1053,25 +1073,53 @@ class CommandHandler:
         receivers = 0
 
         # Construct raw KESP bytes for the broadcast push
-        kesp_payload = (
-            f"A3\n"
-            f"S7\nmessage\n"
-            f"S{len(channel.encode('utf-8'))}\n{channel}\n"
-            f"S{len(message.encode('utf-8'))}\n{message}\n"
-        ).encode("utf-8")
+        kesp_payload = KESPEncoder.encode_command(["message", channel, message])
 
         dead_sockets = []
         for sock in subscribers:
             try:
-                sock.sendall(kesp_payload)
+                self._push(sock, kesp_payload)
                 receivers += 1
             except Exception:
                 dead_sockets.append(sock)
 
         for dead in dead_sockets:
             subscribers.remove(dead)
-
-        return
+        if not subscribers:
+            del self._channels[channel]
+
+        return receivers
+
+    def _push(self, sub, payload: bytes) -> None:
+        """Delivers one pub/sub message to a subscriber, raising if it is gone.
+
+        The server hands subscribers over as asyncio StreamWriters, and it runs
+        commands in worker threads, where a StreamWriter must not be touched
+        directly: the write is queued onto the event loop instead. A plain
+        socket (any object with sendall) is written to directly."""
+        if hasattr(sub, "sendall"):
+            sub.sendall(payload)
+            return
+        if sub.is_closing():
+            raise ConnectionError("subscriber is gone")
+        transport = getattr(sub, "transport", None)
+        if (
+            transport is not None
+            and transport.get_write_buffer_size() > self.MAX_SUBSCRIBER_BUFFER
+        ):
+            # A subscriber that isn't reading would otherwise make the server
+            # hold every message for it, without limit. Cut it off instead.
+            loop = self.event_loop
+            if loop is not None and loop.is_running():
+                loop.call_soon_threadsafe(sub.close)
+            else:
+                sub.close()
+            raise ConnectionError("subscriber output buffer limit exceeded")
+        loop = self.event_loop
+        if loop is not None and loop.is_running():
+            loop.call_soon_threadsafe(sub.write, payload)
+        else:
+            sub.write(payload)
 
     def cmd_slowlog(self, tokens: list[str]) -> Any:
         if len(tokens) < 2: