vs-queue 0.1.4__tar.gz → 0.1.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. {vs_queue-0.1.4/vs_queue.egg-info → vs_queue-0.1.5}/PKG-INFO +9 -1
  2. {vs_queue-0.1.4 → vs_queue-0.1.5}/README.md +8 -0
  3. {vs_queue-0.1.4 → vs_queue-0.1.5}/pyproject.toml +1 -1
  4. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/base/vs_base_queue.py +1 -1
  5. vs_queue-0.1.5/vs_queue/error/vs_message_rejected.py +2 -0
  6. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/manager/vs_queue_manager.py +17 -6
  7. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/provider/vs_rabbitmq_queue.py +13 -6
  8. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/provider/vs_redis_queue.py +46 -18
  9. vs_queue-0.1.5/vs_queue/schema/__init__.py +0 -0
  10. {vs_queue-0.1.4 → vs_queue-0.1.5/vs_queue.egg-info}/PKG-INFO +9 -1
  11. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue.egg-info/SOURCES.txt +2 -0
  12. {vs_queue-0.1.4 → vs_queue-0.1.5}/LICENSE.txt +0 -0
  13. {vs_queue-0.1.4 → vs_queue-0.1.5}/setup.cfg +0 -0
  14. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/__init__.py +0 -0
  15. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/base/__init__.py +0 -0
  16. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/base/vs_base_consumer.py +0 -0
  17. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/decorator/__init__.py +0 -0
  18. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/decorator/vs_queue_listener.py +0 -0
  19. {vs_queue-0.1.4/vs_queue/factory → vs_queue-0.1.5/vs_queue/error}/__init__.py +0 -0
  20. {vs_queue-0.1.4/vs_queue/manager → vs_queue-0.1.5/vs_queue/factory}/__init__.py +0 -0
  21. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/factory/vs_queue_factory.py +0 -0
  22. {vs_queue-0.1.4/vs_queue/provider → vs_queue-0.1.5/vs_queue/manager}/__init__.py +0 -0
  23. {vs_queue-0.1.4/vs_queue/registry → vs_queue-0.1.5/vs_queue/provider}/__init__.py +0 -0
  24. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/py.typed +0 -0
  25. {vs_queue-0.1.4/vs_queue/retry → vs_queue-0.1.5/vs_queue/registry}/__init__.py +0 -0
  26. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/registry/vs_queue_registry.py +0 -0
  27. {vs_queue-0.1.4/vs_queue/schema → vs_queue-0.1.5/vs_queue/retry}/__init__.py +0 -0
  28. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/retry/vs_retry_policy.py +0 -0
  29. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/schema/vs_message.py +0 -0
  30. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue/schema/vs_retention.py +0 -0
  31. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue.egg-info/dependency_links.txt +0 -0
  32. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue.egg-info/requires.txt +0 -0
  33. {vs_queue-0.1.4 → vs_queue-0.1.5}/vs_queue.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: vs-queue
3
- Version: 0.1.4
3
+ Version: 0.1.5
4
4
  Summary: Pluggable async message queue library for Viveka Sutra — Redis Streams, RabbitMQ, retry, dead letter, and annotation-based consumers
5
5
  License: Apache License
6
6
  Version 2.0, January 2004
@@ -1246,3 +1246,11 @@ Decorator. Registers a `VsBaseConsumer` subclass for auto-discovery by `VsQueueM
1246
1246
  ```bash
1247
1247
  ./run_tests.sh
1248
1248
  ```
1249
+
1250
+ ## Agent request/reply integration
1251
+
1252
+ `vs-agent[queue]` provides `VsQueueActionConsumer`, using `VsBaseConsumer`, `VsQueueManager`, `VsMessage`, and the selected provider. Callers place `reply_to`, `correlation_id`, and authentication credentials in message headers. The agent validates/authenticates, executes guards and the action, then publishes a new response before returning for acknowledgement. Response headers never inherit credentials.
1253
+
1254
+ `VsQueueManager.register(queue_name, consumer, retry_policy=...)` forwards retry policies to providers; `register_listeners()` applies the retry settings declared by `@queue_listener`. `VsMessageRejected` marks permanent failures, which bypass retries and are dead-lettered when configured. Retry/DLQ publication failures must not acknowledge the original delivery. Providers overwrite `source_queue` with the actual subscribed queue before retrying; callers cannot redirect retries through that header.
1255
+
1256
+ Redis replays the current consumer's pending entries after a subscription restart and periodically uses XAUTOCLAIM to recover stale deliveries from dead consumers (Redis servers supporting XAUTOCLAIM). Its `credentials["claim_idle_ms"]` option defaults to 300000; set it above the longest expected processing time, or zero to disable reclaim. An overdue message may execute more than once. Durable idempotency is the application's responsibility. Older servers log when stale recovery is unavailable.
@@ -1031,3 +1031,11 @@ Decorator. Registers a `VsBaseConsumer` subclass for auto-discovery by `VsQueueM
1031
1031
  ```bash
1032
1032
  ./run_tests.sh
1033
1033
  ```
1034
+
1035
+ ## Agent request/reply integration
1036
+
1037
+ `vs-agent[queue]` provides `VsQueueActionConsumer`, using `VsBaseConsumer`, `VsQueueManager`, `VsMessage`, and the selected provider. Callers place `reply_to`, `correlation_id`, and authentication credentials in message headers. The agent validates/authenticates, executes guards and the action, then publishes a new response before returning for acknowledgement. Response headers never inherit credentials.
1038
+
1039
+ `VsQueueManager.register(queue_name, consumer, retry_policy=...)` forwards retry policies to providers; `register_listeners()` applies the retry settings declared by `@queue_listener`. `VsMessageRejected` marks permanent failures, which bypass retries and are dead-lettered when configured. Retry/DLQ publication failures must not acknowledge the original delivery. Providers overwrite `source_queue` with the actual subscribed queue before retrying; callers cannot redirect retries through that header.
1040
+
1041
+ Redis replays the current consumer's pending entries after a subscription restart and periodically uses XAUTOCLAIM to recover stale deliveries from dead consumers (Redis servers supporting XAUTOCLAIM). Its `credentials["claim_idle_ms"]` option defaults to 300000; set it above the longest expected processing time, or zero to disable reclaim. An overdue message may execute more than once. Durable idempotency is the application's responsibility. Older servers log when stale recovery is unavailable.
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "vs-queue"
7
- version = "0.1.4"
7
+ version = "0.1.5"
8
8
  description = "Pluggable async message queue library for Viveka Sutra — Redis Streams, RabbitMQ, retry, dead letter, and annotation-based consumers"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.11"
@@ -38,7 +38,7 @@ class VsBaseQueue(ABC):
38
38
  pass
39
39
 
40
40
  @abstractmethod
41
- async def subscribe(self, queue: str, consumer: "VsBaseConsumer") -> None:
41
+ async def subscribe(self, queue: str, consumer: "VsBaseConsumer", retry_policy=None) -> None:
42
42
  pass
43
43
 
44
44
  async def blpop(self, key: str, timeout: int = 30) -> Optional[str]:
@@ -0,0 +1,2 @@
1
+ class VsMessageRejected(Exception):
2
+ """Permanent failure: dead-letter (when configured) rather than retrying."""
@@ -5,6 +5,7 @@ from uuid import uuid4
5
5
 
6
6
  from vs_queue.base.vs_base_consumer import VsBaseConsumer
7
7
  from vs_queue.base.vs_base_queue import VsBaseQueue
8
+ from vs_queue.retry.vs_retry_policy import VsRetryPolicy
8
9
 
9
10
  _logger = logging.getLogger(__name__)
10
11
 
@@ -19,7 +20,7 @@ class VsQueueManager:
19
20
  self._connected = False
20
21
  self._listeners_registered = False
21
22
 
22
- async def register(self, queue_name: str, consumer: VsBaseConsumer) -> str:
23
+ async def register(self, queue_name: str, consumer: VsBaseConsumer, retry_policy: Optional[VsRetryPolicy] = None) -> str:
23
24
  if not self._connected:
24
25
  await self._queue.connect()
25
26
  self._connected = True
@@ -29,7 +30,7 @@ class VsQueueManager:
29
30
  self._consumers[uid] = (queue_name, consumer)
30
31
 
31
32
  task = asyncio.create_task(
32
- self._run(queue_name, consumer),
33
+ self._run(queue_name, consumer, retry_policy),
33
34
  name=f"consumer:{queue_name}:{uid}",
34
35
  )
35
36
  self._tasks[uid] = task
@@ -44,7 +45,11 @@ class VsQueueManager:
44
45
  from vs_queue.decorator.vs_queue_listener import get_registered_listeners
45
46
  for consumer_class, meta in get_registered_listeners():
46
47
  for _ in range(meta.concurrency):
47
- await self.register(meta.queue, consumer_class())
48
+ await self.register(meta.queue, consumer_class(), VsRetryPolicy(
49
+ max_retries=meta.max_retries,
50
+ backoff_seconds=meta.retry_backoff_seconds,
51
+ dead_letter_queue=meta.dead_letter_queue,
52
+ ))
48
53
  _logger.info(f"Listeners registered | queue={meta.queue} concurrency={meta.concurrency}")
49
54
 
50
55
  def get_listeners(self) -> Dict[str, List[VsBaseConsumer]]:
@@ -53,6 +58,10 @@ class VsQueueManager:
53
58
  result.setdefault(queue_name, []).append(consumer)
54
59
  return result
55
60
 
61
+ def has_stopped_listeners(self) -> bool:
62
+ """Whether a registered subscription task has terminated."""
63
+ return any(task.done() for task in self._tasks.values())
64
+
56
65
  async def stop_listener(self, queue_name: str, uuid: Optional[str] = None) -> None:
57
66
  if uuid and uuid in self._consumers:
58
67
  actual_queue, _ = self._consumers[uuid]
@@ -92,14 +101,16 @@ class VsQueueManager:
92
101
  queue_name, _ = self._consumers.pop(uid, (None, None))
93
102
  _logger.info(f"Listener stopped | queue={queue_name} uuid={uid}")
94
103
 
95
- async def _run(self, queue_name: str, consumer: VsBaseConsumer) -> None:
104
+ async def _run(self, queue_name: str, consumer: VsBaseConsumer, retry_policy: Optional[VsRetryPolicy] = None) -> None:
96
105
  attempts = 0
97
106
  while True:
98
107
  try:
99
- await self._queue.subscribe(queue_name, consumer)
108
+ if retry_policy is None:
109
+ await self._queue.subscribe(queue_name, consumer)
110
+ else:
111
+ await self._queue.subscribe(queue_name, consumer, retry_policy=retry_policy)
100
112
  break
101
113
  except asyncio.CancelledError:
102
- await consumer.stop()
103
114
  break
104
115
  except Exception as e:
105
116
  attempts += 1
@@ -6,6 +6,7 @@ from vs_queue.base.vs_base_consumer import VsBaseConsumer
6
6
  from vs_queue.base.vs_base_queue import VsBaseQueue
7
7
  from vs_queue.retry.vs_retry_policy import VsRetryPolicy
8
8
  from vs_queue.schema.vs_message import VsMessage
9
+ from vs_queue.error.vs_message_rejected import VsMessageRejected
9
10
 
10
11
  _logger = logging.getLogger(__name__)
11
12
 
@@ -87,6 +88,7 @@ class VsRabbitMQQueue(VsBaseQueue):
87
88
  headers=data["headers"],
88
89
  payload=data["payload"],
89
90
  )
91
+ message.headers["source_queue"] = queue
90
92
  _logger.info(f"Message received | queue={queue} id={message.id} retry_count={message.retry_count}")
91
93
  try:
92
94
  await consumer.handle(message)
@@ -94,15 +96,20 @@ class VsRabbitMQQueue(VsBaseQueue):
94
96
  _logger.debug(f"Message processed | queue={queue} id={message.id}")
95
97
  except Exception as e:
96
98
  await consumer.on_error(message, e)
97
- if retry_policy:
98
- retried = await retry_policy.handle_failure(message, e, self)
99
- if retried:
99
+ try:
100
+ if isinstance(e, VsMessageRejected):
101
+ if retry_policy and retry_policy.dead_letter_queue:
102
+ await self.publish(retry_policy.dead_letter_queue, message)
103
+ await amqp_message.ack()
104
+ elif retry_policy:
105
+ await retry_policy.handle_failure(message, e, self)
100
106
  await amqp_message.ack()
101
107
  else:
102
108
  await amqp_message.nack(requeue=False)
103
- else:
104
- await amqp_message.nack(requeue=False)
105
- _logger.error(f"Message failed, no retry policy | id={message.id} error={e}")
109
+ _logger.error(f"Message failed, no retry policy | id={message.id} error={e}")
110
+ except Exception:
111
+ await amqp_message.nack(requeue=True)
112
+ raise
106
113
  finally:
107
114
  await consumer.stop()
108
115
  _logger.info(f"Unsubscribed | queue={queue} consumer={type(consumer).__name__}")
@@ -4,15 +4,11 @@ import logging
4
4
  import time
5
5
  from typing import Any, Dict, Optional
6
6
 
7
- import redis.asyncio as aioredis
8
- from redis.exceptions import ConnectionError as RedisConnectionError
9
- from redis.exceptions import ResponseError as RedisResponseError
10
- from redis.exceptions import TimeoutError as RedisTimeoutError
11
-
12
7
  from vs_queue.base.vs_base_consumer import VsBaseConsumer
13
8
  from vs_queue.base.vs_base_queue import VsBaseQueue
14
9
  from vs_queue.retry.vs_retry_policy import VsRetryPolicy
15
10
  from vs_queue.schema.vs_message import VsMessage
11
+ from vs_queue.error.vs_message_rejected import VsMessageRejected
16
12
 
17
13
  _logger = logging.getLogger(__name__)
18
14
 
@@ -24,10 +20,11 @@ class VsRedisQueue(VsBaseQueue):
24
20
 
25
21
  def __init__(self, host: str, port: int, credentials: Optional[Dict[str, Any]] = None):
26
22
  super().__init__(host, port, credentials)
27
- self._client: Optional[aioredis.Redis] = None
23
+ self._client = None
28
24
  self._last_trim: Dict[str, float] = {}
29
25
 
30
26
  async def connect(self) -> None:
27
+ import redis.asyncio as aioredis
31
28
  password = self._credentials.get("password")
32
29
  self._client = aioredis.Redis(
33
30
  host=self._host,
@@ -92,8 +89,11 @@ class VsRedisQueue(VsBaseQueue):
92
89
  return value
93
90
 
94
91
  async def subscribe(self, queue: str, consumer: VsBaseConsumer, retry_policy: Optional[VsRetryPolicy] = None) -> None:
92
+ from redis.exceptions import ConnectionError as RedisConnectionError
93
+ from redis.exceptions import ResponseError as RedisResponseError
94
+ from redis.exceptions import TimeoutError as RedisTimeoutError
95
95
  group = f"{queue}:group"
96
- consumer_name = f"{queue}:consumer"
96
+ consumer_name = f"{queue}:consumer:{consumer.uuid or 'default'}"
97
97
 
98
98
  try:
99
99
  await self._client.xgroup_create(queue, group, id="0", mkstream=True)
@@ -105,16 +105,38 @@ class VsRedisQueue(VsBaseQueue):
105
105
  await consumer.start()
106
106
  _logger.info(f"Subscribed | queue={queue} consumer={type(consumer).__name__}")
107
107
  backoff = _RECONNECT_MIN_BACKOFF
108
+ # Recover this consumer's unacknowledged messages before accepting new ones.
109
+ pending = True
110
+ claim_idle_ms = int(self._credentials.get("claim_idle_ms", 300000))
111
+ claim_cursor = "0-0"
112
+ next_claim = 0.0
108
113
  try:
109
114
  while True:
110
115
  try:
111
- results = await self._client.xreadgroup(
112
- groupname=group,
113
- consumername=consumer_name,
114
- streams={queue: ">"},
115
- count=1,
116
- block=5000,
117
- )
116
+ results = []
117
+ if not pending and claim_idle_ms > 0 and time.monotonic() >= next_claim:
118
+ try:
119
+ claimed = await self._client.xautoclaim(
120
+ queue, group, consumer_name, min_idle_time=claim_idle_ms,
121
+ start_id=claim_cursor, count=1,
122
+ )
123
+ claim_cursor = claimed[0]
124
+ if claimed[1]:
125
+ results = [(queue, claimed[1])]
126
+ except RedisResponseError as e:
127
+ if "unknown command" not in str(e).lower():
128
+ raise
129
+ _logger.warning("Redis does not support XAUTOCLAIM; stale pending recovery is disabled")
130
+ claim_idle_ms = 0
131
+ next_claim = time.monotonic() + 30
132
+ if not results:
133
+ results = await self._client.xreadgroup(
134
+ groupname=group,
135
+ consumername=consumer_name,
136
+ streams={queue: "0" if pending else ">"},
137
+ count=1,
138
+ block=5000,
139
+ )
118
140
  except asyncio.CancelledError:
119
141
  raise
120
142
  except (RedisConnectionError, RedisTimeoutError, OSError) as e:
@@ -127,7 +149,8 @@ class VsRedisQueue(VsBaseQueue):
127
149
  continue
128
150
 
129
151
  backoff = _RECONNECT_MIN_BACKOFF
130
- if not results:
152
+ if not results or not any(messages for _, messages in results):
153
+ pending = False
131
154
  continue
132
155
 
133
156
  for _, messages in results:
@@ -139,17 +162,22 @@ class VsRedisQueue(VsBaseQueue):
139
162
  headers=json.loads(data["headers"]),
140
163
  payload=json.loads(data["payload"]),
141
164
  )
165
+ message.headers["source_queue"] = queue
142
166
  _logger.info(f"Message received | queue={queue} id={message.id} retry_count={message.retry_count}")
143
167
  try:
144
168
  await consumer.handle(message)
145
169
  await self._client.xack(queue, group, msg_id)
146
170
  _logger.debug(f"Message processed | queue={queue} id={message.id}")
171
+ except VsMessageRejected as e:
172
+ await consumer.on_error(message, e)
173
+ if retry_policy and retry_policy.dead_letter_queue:
174
+ await self.publish(retry_policy.dead_letter_queue, message)
175
+ await self._client.xack(queue, group, msg_id)
147
176
  except Exception as e:
148
177
  await consumer.on_error(message, e)
149
178
  if retry_policy:
150
- retried = await retry_policy.handle_failure(message, e, self)
151
- if retried:
152
- await self._client.xack(queue, group, msg_id)
179
+ await retry_policy.handle_failure(message, e, self)
180
+ await self._client.xack(queue, group, msg_id)
153
181
  else:
154
182
  _logger.error(f"Message failed, no retry policy | id={message.id} error={e}")
155
183
  finally:
File without changes
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: vs-queue
3
- Version: 0.1.4
3
+ Version: 0.1.5
4
4
  Summary: Pluggable async message queue library for Viveka Sutra — Redis Streams, RabbitMQ, retry, dead letter, and annotation-based consumers
5
5
  License: Apache License
6
6
  Version 2.0, January 2004
@@ -1246,3 +1246,11 @@ Decorator. Registers a `VsBaseConsumer` subclass for auto-discovery by `VsQueueM
1246
1246
  ```bash
1247
1247
  ./run_tests.sh
1248
1248
  ```
1249
+
1250
+ ## Agent request/reply integration
1251
+
1252
+ `vs-agent[queue]` provides `VsQueueActionConsumer`, using `VsBaseConsumer`, `VsQueueManager`, `VsMessage`, and the selected provider. Callers place `reply_to`, `correlation_id`, and authentication credentials in message headers. The agent validates/authenticates, executes guards and the action, then publishes a new response before returning for acknowledgement. Response headers never inherit credentials.
1253
+
1254
+ `VsQueueManager.register(queue_name, consumer, retry_policy=...)` forwards retry policies to providers; `register_listeners()` applies the retry settings declared by `@queue_listener`. `VsMessageRejected` marks permanent failures, which bypass retries and are dead-lettered when configured. Retry/DLQ publication failures must not acknowledge the original delivery. Providers overwrite `source_queue` with the actual subscribed queue before retrying; callers cannot redirect retries through that header.
1255
+
1256
+ Redis replays the current consumer's pending entries after a subscription restart and periodically uses XAUTOCLAIM to recover stale deliveries from dead consumers (Redis servers supporting XAUTOCLAIM). Its `credentials["claim_idle_ms"]` option defaults to 300000; set it above the longest expected processing time, or zero to disable reclaim. An overdue message may execute more than once. Durable idempotency is the application's responsibility. Older servers log when stale recovery is unavailable.
@@ -13,6 +13,8 @@ vs_queue/base/vs_base_consumer.py
13
13
  vs_queue/base/vs_base_queue.py
14
14
  vs_queue/decorator/__init__.py
15
15
  vs_queue/decorator/vs_queue_listener.py
16
+ vs_queue/error/__init__.py
17
+ vs_queue/error/vs_message_rejected.py
16
18
  vs_queue/factory/__init__.py
17
19
  vs_queue/factory/vs_queue_factory.py
18
20
  vs_queue/manager/__init__.py
File without changes
File without changes
File without changes
File without changes