kedis-python 0.1.3__tar.gz → 0.2.0__tar.gz

{kedis_python-0.1.3/kedis_python.egg-info → kedis_python-0.2.0}/PKG-INFO

@@ -1,10 +1,16 @@
 Metadata-Version: 2.4
 Name: kedis-python
-Version: 0.1.3
+Version: 0.2.0
 Summary: A Redis-inspired in-memory store in Python.
 Author: V SS Karthik
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/Hogwarts-coder10/kedis-python
+Keywords: redis,in-memory,key-value,database,streams,pub-sub
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Database
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -43,15 +49,17 @@ Rather than focusing on Redis compatibility, Kedis focuses on understanding the
 ## 📚 Engine Documentation
 
 Dive into the internal grimoire to understand how Kedis achieves microsecond latency:
-* [Architecture & Pipeline](docs/architecture.md)
-* [KESP Wire Protocol](docs/protocol.md)
-* [Networking & Surge Tank](docs/networking.md)
-* [Persistence & I/O Drivetrain](docs/persistence.md)
-* [Command Spellbook](docs/commands.md)
-* [TTL & Lifespan Mechanics](docs/ttl.md)
-* [Pub/Sub Broadcasts](docs/pubsub.md)
-* [Benchmarks & Telemetry](docs/benchmarks.md)
-* [Future Roadmap](docs/roadmap.md)
+* [Architecture & Pipeline](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/architecture.md)
+* [KESP Wire Protocol](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/protocol.md)
+* [Networking & Surge Tank](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/networking.md)
+* [Limits & Budgets](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/limits.md)
+* [Persistence & I/O Drivetrain](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/persistence.md)
+* [Command Spellbook](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/commands.md)
+* [TTL & Lifespan Mechanics](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/ttl.md)
+* [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md)
+* [Pub/Sub Broadcasts](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/pubsub.md)
+* [Benchmarks & Telemetry](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/benchmarks.md)
+* [Future Roadmap](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/roadmap.md)
 
 ---
 
@@ -107,6 +115,20 @@ ZRANGE leaderboard
 
 Implemented using a custom Skip List inspired by Redis sorted set internals, with rank-aware span tracking for fast positional queries.
 
+### Streams & Consumer Groups
+
+```bash
+XADD orders * item book qty 2
+XREAD BLOCK 5000 STREAMS orders $
+
+XGROUP CREATE orders shipping 0
+XREADGROUP GROUP shipping worker1 COUNT 10 STREAMS orders >
+XPENDING orders shipping
+XACK orders shipping 1718000000000-0
+```
+
+Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md).
+
 ---
 
 ## ⏳ Expiration Support
@@ -290,6 +312,7 @@ The project includes automated tests covering:
 * data structures
 * persistence recovery
 * expiration behavior
+* streams and consumer groups, from the data structure up to live TCP sessions
 
 Additional coverage is actively being expanded as the project evolves.
 
@@ -361,6 +384,7 @@ Current limitations include:
 * limited fault tolerance compared to production databases
 * no memory-based eviction ceiling (LRU tracks usage, but isn't yet tied to a hard memory limit)
 * AOF persistence is not buffered — writes can trigger a sync depending on config
+* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md#-known-limits)
 
 These limitations are intentional learning opportunities and areas of active development.
 

kedis_python-0.1.3/PKG-INFO → kedis_python-0.2.0/README.md

@@ -1,17 +1,3 @@
-Metadata-Version: 2.4
-Name: kedis-python
-Version: 0.1.3
-Summary: A Redis-inspired in-memory store in Python.
-Author: V SS Karthik
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/Hogwarts-coder10/kedis-python
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: rich
-Requires-Dist: pyfiglet
-Dynamic: license-file
-
 # 🚀 Kedis-Python
 
 **A Redis-inspired in-memory datastore built in pure Python to explore storage engines, networking, persistence, caching, and systems architecture.**
@@ -43,15 +29,17 @@ Rather than focusing on Redis compatibility, Kedis focuses on understanding the
 ## 📚 Engine Documentation
 
 Dive into the internal grimoire to understand how Kedis achieves microsecond latency:
-* [Architecture & Pipeline](docs/architecture.md)
-* [KESP Wire Protocol](docs/protocol.md)
-* [Networking & Surge Tank](docs/networking.md)
-* [Persistence & I/O Drivetrain](docs/persistence.md)
-* [Command Spellbook](docs/commands.md)
-* [TTL & Lifespan Mechanics](docs/ttl.md)
-* [Pub/Sub Broadcasts](docs/pubsub.md)
-* [Benchmarks & Telemetry](docs/benchmarks.md)
-* [Future Roadmap](docs/roadmap.md)
+* [Architecture & Pipeline](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/architecture.md)
+* [KESP Wire Protocol](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/protocol.md)
+* [Networking & Surge Tank](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/networking.md)
+* [Limits & Budgets](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/limits.md)
+* [Persistence & I/O Drivetrain](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/persistence.md)
+* [Command Spellbook](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/commands.md)
+* [TTL & Lifespan Mechanics](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/ttl.md)
+* [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md)
+* [Pub/Sub Broadcasts](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/pubsub.md)
+* [Benchmarks & Telemetry](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/benchmarks.md)
+* [Future Roadmap](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/roadmap.md)
 
 ---
 
@@ -107,6 +95,20 @@ ZRANGE leaderboard
 
 Implemented using a custom Skip List inspired by Redis sorted set internals, with rank-aware span tracking for fast positional queries.
 
+### Streams & Consumer Groups
+
+```bash
+XADD orders * item book qty 2
+XREAD BLOCK 5000 STREAMS orders $
+
+XGROUP CREATE orders shipping 0
+XREADGROUP GROUP shipping worker1 COUNT 10 STREAMS orders >
+XPENDING orders shipping
+XACK orders shipping 1718000000000-0
+```
+
+Append-only event logs with blocking reads, and consumer groups that share a stream between workers with a pending list and acknowledgements. Streams persist through the AOF, `COMPACT` and snapshots, and replicate with the exact IDs and group state the master chose. See [Streams & Consumer Groups](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md).
+
 ---
 
 ## ⏳ Expiration Support
@@ -290,6 +292,7 @@ The project includes automated tests covering:
 * data structures
 * persistence recovery
 * expiration behavior
+* streams and consumer groups, from the data structure up to live TCP sessions
 
 Additional coverage is actively being expanded as the project evolves.
 
@@ -361,6 +364,7 @@ Current limitations include:
 * limited fault tolerance compared to production databases
 * no memory-based eviction ceiling (LRU tracks usage, but isn't yet tied to a hard memory limit)
 * AOF persistence is not buffered — writes can trigger a sync depending on config
+* consumer groups have no `XCLAIM`/`XAUTOCLAIM` yet; the full list of stream limits is in [docs/streams.md](https://github.com/Hogwarts-coder10/kedis-python/blob/main/docs/streams.md#-known-limits)
 
 These limitations are intentional learning opportunities and areas of active development.
 

kedis_python-0.2.0/kedis_python/aof.py

@@ -0,0 +1,138 @@
+"""On-disk record format for the append-only file (AOF).
+
+Every logged command is one KESP-framed record, the same shape the wire
+protocol uses for commands:
+
+    A<count>\\n  then, for each token:  S<length>\\n<payload>\\n
+
+Lengths count bytes of the UTF-8 payload, so a token can hold anything: spaces,
+newlines, tabs, an empty string, text that looks like another command. The old
+format joined tokens with spaces on one line, so a value could not survive a
+restart and a value containing a newline could plant commands in the log.
+
+Strings are encoded with "surrogatepass", so every Python str round-trips
+exactly, lone surrogates included.
+
+Files written by older versions hold one whitespace-separated command per line.
+scan() still reads those, line by line, and a file may mix both: old lines
+first, framed records after.
+"""
+
+import re
+
+_COUNT = re.compile(rb"(?:0|[1-9][0-9]*)")
+MAX_HEADER_LINE = 32  # "A<n>" / "S<n>" lines are never longer than this
+
+
+def encode_record(tokens) -> bytes:
+    """One command as a framed record."""
+    parts = [b"A%d\n" % len(tokens)]
+    for token in tokens:
+        raw = str(token).encode("utf-8", "surrogatepass")
+        parts.append(b"S%d\n" % len(raw))
+        parts.append(raw)
+        parts.append(b"\n")
+    return b"".join(parts)
+
+
+class ScanResult:
+    """How a scan ended.
+
+    status:   "ok"      every byte was consumed
+              "torn"    the file ends in the middle of a record (a crash while
+                        writing); everything before good_end is intact
+              "corrupt" a record is malformed; see reason
+    good_end: offset just past the last complete record or line
+    """
+
+    __slots__ = ("good_end", "reason", "status")
+
+    def __init__(self):
+        self.status = "ok"
+        self.good_end = 0
+        self.reason = ""
+
+
+def _count(buf, start: int, end: int):
+    field = bytes(buf[start:end])
+    return int(field) if _COUNT.fullmatch(field) else None
+
+
+def scan(buf, result: ScanResult):
+    """Yields (tokens, end_offset) for each command in buf, a bytes-like object
+    that supports find() and slicing (bytes or mmap). Stops at the first
+    incomplete or malformed record and says why in result."""
+    total = len(buf)
+    pos = 0
+
+    def stop(status: str, reason: str):
+        result.status = status
+        result.reason = reason
+
+    while pos < total:
+        if buf[pos : pos + 1] == b"A":
+            following = buf[pos + 1 : pos + 2]
+            if following == b"":
+                return stop("torn", "file ends inside a record header")
+            # A legacy command word starts "A" + a letter (AUTH, ...). Anything
+            # else after the "A" is a record header and must validate as one.
+            framed = not following.isalpha()
+        else:
+            framed = False
+
+        if not framed:
+            # Legacy line: whitespace-separated tokens, as older versions wrote.
+            nl = buf.find(b"\n", pos)
+            line_end = total if nl == -1 else nl
+            next_pos = total if nl == -1 else nl + 1
+            tokens = bytes(buf[pos:line_end]).decode("utf-8", "replace").split()
+            result.good_end = next_pos
+            if tokens:
+                yield tokens, next_pos
+            pos = next_pos
+            continue
+
+        nl = buf.find(b"\n", pos)
+        if nl == -1:
+            if total - pos > MAX_HEADER_LINE:
+                return stop("corrupt", "array header too long")
+            return stop("torn", "file ends inside a record header")
+        if nl - pos > MAX_HEADER_LINE:
+            return stop("corrupt", "array header too long")
+        count = _count(buf, pos + 1, nl)
+        if count is None:
+            return stop("corrupt", "invalid array header")
+
+        cursor = nl + 1
+        tokens = []
+        for _ in range(count):
+            if cursor >= total:
+                return stop("torn", "file ends between tokens")
+            if buf[cursor : cursor + 1] != b"S":
+                return stop("corrupt", "expected a string header")
+            nl = buf.find(b"\n", cursor)
+            if nl == -1:
+                if total - cursor > MAX_HEADER_LINE:
+                    return stop("corrupt", "string header too long")
+                return stop("torn", "file ends inside a string header")
+            if nl - cursor > MAX_HEADER_LINE:
+                return stop("corrupt", "string header too long")
+            length = _count(buf, cursor + 1, nl)
+            if length is None:
+                return stop("corrupt", "invalid string length")
+            start = nl + 1
+            end = start + length
+            if end + 1 > total:
+                return stop("torn", "file ends inside a string")
+            if buf[end : end + 1] != b"\n":
+                return stop("corrupt", "string is not newline-terminated")
+            try:
+                tokens.append(bytes(buf[start:end]).decode("utf-8", "surrogatepass"))
+            except UnicodeDecodeError:
+                return stop("corrupt", "string is not valid UTF-8")
+            cursor = end + 1
+
+        result.good_end = cursor
+        if tokens:
+            yield tokens, cursor
+        pos = cursor