redis-stream-queue 0.1.0__tar.gz

redis_stream_queue-0.1.0/.gitignore

@@ -0,0 +1,15 @@
+.venv/
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+htmlcov/
+.coverage
+examples/*.env
+.claude
+**/__pycache__
+*.pyc
+.idea
+.vsc

redis_stream_queue-0.1.0/.run/basic_worker.run.xml

@@ -0,0 +1,26 @@
+<component name="ProjectRunConfigurationManager">
+  <configuration default="false" name="basic_worker" type="PythonConfigurationType" factoryName="Python" nameIsGenerated="true">
+    <module name="redis-stream-queue" />
+    <option name="ENV_FILES" value="" />
+    <option name="INTERPRETER_OPTIONS" value="" />
+    <option name="PARENT_ENVS" value="true" />
+    <envs>
+      <env name="PYTHONUNBUFFERED" value="1" />
+    </envs>
+    <option name="SDK_HOME" value="" />
+    <option name="WORKING_DIRECTORY" value="$PROJECT_DIR$/examples" />
+    <option name="IS_MODULE_SDK" value="true" />
+    <option name="ADD_CONTENT_ROOTS" value="true" />
+    <option name="ADD_SOURCE_ROOTS" value="true" />
+    <EXTENSION ID="PythonCoverageRunConfigurationExtension" runner="coverage.py" />
+    <option name="RUN_TOOL" value="" />
+    <option name="SCRIPT_NAME" value="$PROJECT_DIR$/examples/basic_worker.py" />
+    <option name="PARAMETERS" value="" />
+    <option name="SHOW_COMMAND_LINE" value="false" />
+    <option name="EMULATE_TERMINAL" value="false" />
+    <option name="MODULE_MODE" value="false" />
+    <option name="REDIRECT_INPUT" value="false" />
+    <option name="INPUT_FILE" value="" />
+    <method v="2" />
+  </configuration>
+</component>

redis_stream_queue-0.1.0/CLAUDE.md

@@ -0,0 +1,65 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+```bash
+# Install (editable + dev deps)
+pip install -e ".[dev]"
+
+# Run all tests (no real Redis needed — uses fakeredis)
+pytest tests/
+
+# Single file
+pytest tests/test_consumer.py
+
+# Single test
+pytest tests/test_consumer.py::test_poison_pill_goes_to_dlq -v -s
+
+# With coverage
+pytest tests/ --cov=src/redis_stream_queue --cov-report=term-missing
+
+# Build distribution
+hatch build
+
+# Run example (requires Redis on localhost:6379)
+python examples/basic_worker.py
+```
+
+## Architecture
+
+Single-package library (`src/redis_stream_queue/`) with these layers:
+
+**`client.py` — `StreamClient`**  
+Thin wrapper over `redis.asyncio`. Owns the connection pool. Exposes raw Redis Stream commands (`XADD`, `XREADGROUP`, `XAUTOCLAIM`, `XACK`, `XPENDING`, `XRANGE`, `XGROUP CREATE`). All other classes go through this. Factory methods: `from_cluster()`, `from_url()`.
+
+**`group.py` — `ConsumerConfig` + `ConsumerGroup`**  
+`ConsumerConfig` is a dataclass holding all tuning knobs. `ConsumerGroup` holds a class-level `_known: set` registry to avoid redundant `XGROUP CREATE` calls across iterations — entries are removed on `NOGROUP` error so the next `ensure()` re-creates the group. Also handles `stats()`, `health_check()`, `pending_details()`.
+
+**`consumer.py` — `StreamConsumer`**  
+The main loop (`run()` → `run_once()`). Four-step iteration: ensure group → XREADGROUP → XAUTOCLAIM cursor loop → poison-pill sweep. Maintains a `weakref.WeakSet` class-level `_instances` registry for `all_metrics()`. Tracks `ConsumerMetrics` via `_TpsTracker` (sliding 60 s window).
+
+**`producer.py` — `StreamProducer`**  
+`push(data)` → serialize → `XADD MAXLEN`. Same weakref registry pattern as consumer for `all_metrics()`. Tracks `ProducerMetrics`.
+
+**`retry.py` — `RetryHandler`**  
+Called by `StreamConsumer._process_batch()` for decode errors (immediate DLQ + XACK, no retry) and by `handle_poison_pills()` for messages exceeding `max_deliveries` (XPENDING_RANGE → XRANGE per ID → `dlq_handler` → batched XACK).
+
+**`message.py`**  
+Data classes only: `StreamMessage`, `PendingEntry`, `StreamStats`, `ConsumerInfo`, `ConsumerMetrics`, `ProducerMetrics`, `_TpsTracker`.
+
+**`serializers.py`**  
+Protocol `Serializer` + `JsonSerializer` (default), `MsgpackSerializer` (optional dep), `PickleSerializer`. Decode failures always go to DLQ — never retried.
+
+## Testing
+
+Tests use `fakeredis` — no real Redis instance required. Two `autouse` fixtures in `conftest.py` reset class-level state between tests: `ConsumerGroup._known` and the `WeakSet` instance registries on both `StreamConsumer` and `StreamProducer`. The `make_client` fixture monkeypatches `_get_client()` to return a `FakeRedis` instance.
+
+## Key invariants
+
+- `handler` must return `list[str]` of ACKed IDs. Returning `None` logs a warning and skips ACK; returning `[]` leaves all messages in PEL for retry.
+- Serializer must match between producer and consumer — mismatches become decode errors routed to DLQ.
+- Redis Cluster requires stream and DLQ stream to share a hash tag (e.g. `{orders}_main` / `{orders}_dlq`).
+- `all_metrics()` is in-process only — it does not aggregate across pods.
+- `min_idle_claim_ms` should be ≥ 2× maximum handler latency to avoid premature cross-pod reclaim.

redis_stream_queue-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Doan Vu Van
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.