taskiq-redis 1.0.2__py3-none-any.whl → 1.0.4__py3-none-any.whl

taskiq_redis-1.0.4.dist-info/METADATA

@@ -0,0 +1,215 @@
+Metadata-Version: 2.3
+Name: taskiq-redis
+Version: 1.0.4
+Summary: Redis integration for taskiq
+Keywords: taskiq,tasks,distributed,async,redis,result_backend
+Author: taskiq-team
+Author-email: taskiq@norely.com
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.8
+Requires-Dist: redis (>=5,<6)
+Requires-Dist: taskiq (>=0.11.12,<1)
+Project-URL: Homepage, https://github.com/taskiq-python/taskiq-redis
+Project-URL: Repository, https://github.com/taskiq-python/taskiq-redis
+Description-Content-Type: text/markdown
+
+# TaskIQ-Redis
+
+Taskiq-redis is a plugin for taskiq that adds a new broker and result backend based on redis.
+
+# Installation
+
+To use this project you must have installed core taskiq library:
+```bash
+pip install taskiq
+```
+This project can be installed using pip:
+```bash
+pip install taskiq-redis
+```
+
+# Usage
+
+Let's see the example with the redis broker and redis async result:
+
+```python
+# broker.py
+import asyncio
+
+from taskiq_redis import RedisAsyncResultBackend, RedisStreamBroker
+
+result_backend = RedisAsyncResultBackend(
+    redis_url="redis://localhost:6379",
+)
+
+# Or you can use PubSubBroker if you need broadcasting
+# Or ListQueueBroker if you don't want acknowledges
+broker = RedisStreamBroker(
+    url="redis://localhost:6379",
+).with_result_backend(result_backend)
+
+
+@broker.task
+async def best_task_ever() -> None:
+    """Solve all problems in the world."""
+    await asyncio.sleep(5.5)
+    print("All problems are solved!")
+
+
+async def main():
+    task = await best_task_ever.kiq()
+    print(await task.wait_result())
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+Launch the workers:
+`taskiq worker broker:broker`
+Then run the main code:
+`python3 broker.py`
+
+
+## Brokers
+
+This package contains 6 broker implementations.
+3 broker types:
+* PubSub broker
+* ListQueue broker
+* Stream broker
+
+Each of type is implemented for each redis architecture:
+* Single node
+* Cluster
+* Sentinel
+
+Here's a small breakdown of how they differ from eachother.
+
+
+### PubSub
+
+By default on old redis versions PUBSUB was the way of making redis into a queue.
+But using PUBSUB means that all messages delivered to all subscribed consumers.
+
+> [!WARNING]
+> This broker doesn't support acknowledgements. If during message processing
+> Worker was suddenly killed the message is going to be lost.
+
+### ListQueue
+
+This broker creates a list of messages at some key. Adding new tasks will be done
+by appending them from the left side using `lpush`, and taking them from the right side using `brpop`.
+
+> [!WARNING]
+> This broker doesn't support acknowledgements. If during message processing
+> Worker was suddenly killed the message is going to be lost.
+
+### Stream
+
+Stream brokers use redis [stream type](https://redis.io/docs/latest/develop/data-types/streams/) to store and fetch messages.
+
+> [!TIP]
+> This broker **supports** acknowledgements and therefore is fine to use in cases when data durability is
+> required.
+
+## RedisAsyncResultBackend configuration
+
+RedisAsyncResultBackend parameters:
+* `redis_url` - url to redis.
+* `keep_results` - flag to not remove results from Redis after reading.
+* `result_ex_time` - expire time in seconds (by default - not specified)
+* `result_px_time` - expire time in milliseconds (by default - not specified)
+* Any other keyword arguments are passed to `redis.asyncio.BlockingConnectionPool`.
+  Notably, you can use `timeout` to set custom timeout in seconds for reconnects
+  (or set it to `None` to try reconnects indefinitely).
+
+> [!WARNING]
+> **It is highly recommended to use expire time in RedisAsyncResultBackend**
+> If you want to add expiration, either `result_ex_time` or `result_px_time` must be set.
+> ```python
+> # First variant
+> redis_async_result = RedisAsyncResultBackend(
+>     redis_url="redis://localhost:6379",
+>     result_ex_time=1000,
+> )
+>
+> # Second variant
+> redis_async_result = RedisAsyncResultBackend(
+>     redis_url="redis://localhost:6379",
+>     result_px_time=1000000,
+> )
+> ```
+
+
+## Schedule sources
+
+
+You can use this package to add dynamic schedule sources. They are used to store
+schedules for taskiq scheduler.
+
+The advantage of using schedule sources from this package over default `LabelBased` source is that you can
+dynamically add schedules in it.
+
+We have two types of schedules:
+
+* `RedisScheduleSource`
+* `ListRedisScheduleSource`
+
+
+### RedisScheduleSource
+
+This source is super simple. It stores all schedules by key `{prefix}:{schedule_id}`. When scheduler requests
+schedules, it retrieves all values from redis that start with a given `prefix`.
+
+This is very ineficent and should not be used for high-volume schedules. Because if you have `1000` schedules, this scheduler will make at least `20` requests to retrieve them (we use `scan` and `mget` to minimize number of calls).
+
+### ListRedisScheduleSource
+
+This source holds values in lists.
+
+* For cron tasks it uses key `{prefix}:cron`.
+* For timed schedules it uses key `{prefix}:time:{time}` where `{time}` is actually time where schedules should run.
+
+The main advantage of this approach is that we only fetch tasks we need to run at a given time and do not perform any excesive calls to redis.
+
+
+### Migration from one source to another
+
+To migrate from `RedisScheduleSource` to `ListRedisScheduleSource` you can define the latter as this:
+
+```python
+# broker.py
+import asyncio
+import datetime
+
+from taskiq import TaskiqScheduler
+
+from taskiq_redis import ListRedisScheduleSource, RedisStreamBroker
+from taskiq_redis.schedule_source import RedisScheduleSource
+
+broker = RedisStreamBroker(url="redis://localhost:6379")
+
+old_source = RedisScheduleSource("redis://localhost/1", prefix="prefix1")
+array_source = ListRedisScheduleSource(
+    "redis://localhost/1",
+    prefix="prefix2",
+    # To migrate schedules from an old source.
+).with_migrate_from(
+    old_source,
+    # To delete schedules from an old source.
+    delete_schedules=True,
+)
+scheduler = TaskiqScheduler(broker, [array_source])
+```
+
+During startup the scheduler will try to migrate schedules from an old source to a new one. Please be sure to specify different prefixe just to avoid any kind of collision between these two.
+

taskiq_redis-1.0.4.dist-info/RECORD

@@ -0,0 +1,13 @@
+taskiq_redis/__init__.py,sha256=Sl4m9rKxweU1t0m289Qtf0qm4xSSkkFHoOfKq6qaz6g,1192
+taskiq_redis/exceptions.py,sha256=7buBJ7CRVWd5WqVqSjtHO8cVL7QzZg-DOM3nB87t-Sk,738
+taskiq_redis/list_schedule_source.py,sha256=guWql2hs2WT35vZtrsW1W9-TvaHsX5Lq_CyRjrl0tGA,9458
+taskiq_redis/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+taskiq_redis/redis_backend.py,sha256=MLBaUN3Zx-DLvm1r-lgPU82_WZq9kc6oTxYI8LQjd6k,19882
+taskiq_redis/redis_broker.py,sha256=ZLn7LAHj8Sh_oyW5hMgD7PZPQfUdXNPKdqhBcr9Okmg,9775
+taskiq_redis/redis_cluster_broker.py,sha256=FuWl5fP7Fwr9FbytErmhcUGjRCdPexDK2Co2u6kpDlo,6591
+taskiq_redis/redis_sentinel_broker.py,sha256=wHnbG3xuD_ruhhwp4AXo91NNjq8v2iufUZ0i_HbBRVQ,9073
+taskiq_redis/schedule_source.py,sha256=hqpcs2D8W90KUDHREKblisnhGCE9dbVOtKtuJcOTGZw,9915
+taskiq_redis-1.0.4.dist-info/LICENSE,sha256=lEHEEE-ZxmuItxYgUMPiFWdRcAITxE8DFMNyAg4eOYE,1075
+taskiq_redis-1.0.4.dist-info/METADATA,sha256=sv_06NsLK3SODn9rj404w-mGKpnIWrI5iGLZEPyaBj8,6573
+taskiq_redis-1.0.4.dist-info/WHEEL,sha256=fGIA9gx4Qxk2KDKeNJCbOEwSrmLtjWCwzBz351GyrPQ,88
+taskiq_redis-1.0.4.dist-info/RECORD,,

{taskiq_redis-1.0.2.dist-info → taskiq_redis-1.0.4.dist-info}/WHEEL

@@ -1,4 +1,4 @@
 Wheel-Version: 1.0
-Generator: poetry-core 1.9.0
+Generator: poetry-core 2.1.2
 Root-Is-Purelib: true
 Tag: py3-none-any

taskiq_redis-1.0.2.dist-info/METADATA

@@ -1,125 +0,0 @@
-Metadata-Version: 2.1
-Name: taskiq-redis
-Version: 1.0.2
-Summary: Redis integration for taskiq
-Home-page: https://github.com/taskiq-python/taskiq-redis
-Keywords: taskiq,tasks,distributed,async,redis,result_backend
-Author: taskiq-team
-Author-email: taskiq@norely.com
-Requires-Python: >=3.8.1,<4.0.0
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.8
-Requires-Dist: redis (>=5,<6)
-Requires-Dist: taskiq (>=0.11.1,<1)
-Project-URL: Repository, https://github.com/taskiq-python/taskiq-redis
-Description-Content-Type: text/markdown
-
-# TaskIQ-Redis
-
-Taskiq-redis is a plugin for taskiq that adds a new broker and result backend based on redis.
-
-# Installation
-
-To use this project you must have installed core taskiq library:
-```bash
-pip install taskiq
-```
-This project can be installed using pip:
-```bash
-pip install taskiq-redis
-```
-
-# Usage
-
-Let's see the example with the redis broker and redis async result:
-
-```python
-# broker.py
-import asyncio
-
-from taskiq_redis import ListQueueBroker, RedisAsyncResultBackend
-
-redis_async_result = RedisAsyncResultBackend(
-    redis_url="redis://localhost:6379",
-)
-
-# Or you can use PubSubBroker if you need broadcasting
-broker = ListQueueBroker(
-    url="redis://localhost:6379",
-    result_backend=redis_async_result,
-)
-
-
-@broker.task
-async def best_task_ever() -> None:
-    """Solve all problems in the world."""
-    await asyncio.sleep(5.5)
-    print("All problems are solved!")
-
-
-async def main():
-    task = await best_task_ever.kiq()
-    print(await task.wait_result())
-
-
-if __name__ == "__main__":
-    asyncio.run(main())
-```
-
-Launch the workers:
-`taskiq worker broker:broker`
-Then run the main code:
-`python3 broker.py`
-
-## PubSubBroker and ListQueueBroker configuration
-
-We have two brokers with similar interfaces, but with different logic.
-The PubSubBroker uses redis' pubsub mechanism and is very powerful,
-but it executes every task on all workers, because PUBSUB broadcasts message
-to all subscribers.
-
-If you want your messages to be processed only once, please use ListQueueBroker.
-It uses redis' [LPUSH](https://redis.io/commands/lpush/) and [BRPOP](https://redis.io/commands/brpop/) commands to deal with messages.
-
-Brokers parameters:
-* `url` - url to redis.
-* `task_id_generator` - custom task_id genertaor.
-* `result_backend` - custom result backend.
-* `queue_name` - name of the pub/sub channel in redis.
-* `max_connection_pool_size` - maximum number of connections in pool.
-* Any other keyword arguments are passed to `redis.asyncio.BlockingConnectionPool`.
-  Notably, you can use `timeout` to set custom timeout in seconds for reconnects
-  (or set it to `None` to try reconnects indefinitely).
-
-## RedisAsyncResultBackend configuration
-
-RedisAsyncResultBackend parameters:
-* `redis_url` - url to redis.
-* `keep_results` - flag to not remove results from Redis after reading.
-* `result_ex_time` - expire time in seconds (by default - not specified)
-* `result_px_time` - expire time in milliseconds (by default - not specified)
-* Any other keyword arguments are passed to `redis.asyncio.BlockingConnectionPool`.
-  Notably, you can use `timeout` to set custom timeout in seconds for reconnects
-  (or set it to `None` to try reconnects indefinitely).
-> IMPORTANT: **It is highly recommended to use expire time ​​in RedisAsyncResultBackend**
-> If you want to add expiration, either `result_ex_time` or `result_px_time` must be set.
->```python
-># First variant
->redis_async_result = RedisAsyncResultBackend(
->    redis_url="redis://localhost:6379",
->    result_ex_time=1000,
->)
->
-># Second variant
->redis_async_result = RedisAsyncResultBackend(
->    redis_url="redis://localhost:6379",
->    result_px_time=1000000,
->)
->```
-

taskiq_redis-1.0.2.dist-info/RECORD

@@ -1,12 +0,0 @@
-taskiq_redis/__init__.py,sha256=UEW3rQXt4jinMnAKJlpXQhyPDh6SU2in0bPgzfIo3y4,911
-taskiq_redis/exceptions.py,sha256=eS4bfZVAjyMsnFs3IF74uYwO1KZOlrYxhxgPqD49ztU,561
-taskiq_redis/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-taskiq_redis/redis_backend.py,sha256=mGU3rXJ727X-qYqSfaderTFVGz93NpCyHCf5PDFTjGk,19543
-taskiq_redis/redis_broker.py,sha256=JeoA3-quZYqa_wixJefMRYPkZe94x-qoQb6tQKkHLzg,4733
-taskiq_redis/redis_cluster_broker.py,sha256=CgPKkoEHZ1moNM-VNmzPQdjjNOrhiVUCNV-7FrUgqTo,2121
-taskiq_redis/redis_sentinel_broker.py,sha256=5MxUFIX7qRyDT7IHebLUhxAmmUwk1_b2sxjpSXRcjlo,4114
-taskiq_redis/schedule_source.py,sha256=bk96UBg8op-Xqg_PVETgyDb92cDaY69EAjpP8GvYSnY,10068
-taskiq_redis-1.0.2.dist-info/LICENSE,sha256=lEHEEE-ZxmuItxYgUMPiFWdRcAITxE8DFMNyAg4eOYE,1075
-taskiq_redis-1.0.2.dist-info/METADATA,sha256=6A_nDPLAmO92y_Db7vNUIGiOGqH7gTm_rCpb0KMIPOc,4030
-taskiq_redis-1.0.2.dist-info/WHEEL,sha256=sP946D7jFCHeNz5Iq4fL4Lu-PrWrFsgfLXbbkciIZwg,88
-taskiq_redis-1.0.2.dist-info/RECORD,,