redis-allocator 0.0.1__py3-none-any.whl → 0.3.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- redis_allocator/__init__.py +5 -1
- redis_allocator/_version.py +1 -0
- redis_allocator/allocator.py +816 -280
- redis_allocator/lock.py +66 -17
- redis_allocator/task_queue.py +81 -57
- redis_allocator-0.3.1.dist-info/METADATA +529 -0
- redis_allocator-0.3.1.dist-info/RECORD +15 -0
- {redis_allocator-0.0.1.dist-info → redis_allocator-0.3.1.dist-info}/licenses/LICENSE +21 -21
- tests/conftest.py +160 -46
- tests/test_allocator.py +461 -488
- tests/test_lock.py +675 -338
- tests/test_task_queue.py +136 -136
- redis_allocator-0.0.1.dist-info/METADATA +0 -229
- redis_allocator-0.0.1.dist-info/RECORD +0 -14
- {redis_allocator-0.0.1.dist-info → redis_allocator-0.3.1.dist-info}/WHEEL +0 -0
- {redis_allocator-0.0.1.dist-info → redis_allocator-0.3.1.dist-info}/top_level.txt +0 -0
@@ -0,0 +1,529 @@
|
|
1
|
+
Metadata-Version: 2.4
|
2
|
+
Name: redis-allocator
|
3
|
+
Version: 0.3.1
|
4
|
+
Summary: Redis-based resource allocation system.
|
5
|
+
Home-page: https://github.com/invoker-bot/RedisAllocator-python
|
6
|
+
Author: Invoker Bot
|
7
|
+
Author-email: invoker-bot@outlook.com
|
8
|
+
License: MIT
|
9
|
+
Classifier: Development Status :: 4 - Beta
|
10
|
+
Classifier: Intended Audience :: Developers
|
11
|
+
Classifier: Programming Language :: Python :: 3.9
|
12
|
+
Classifier: License :: OSI Approved :: MIT License
|
13
|
+
Classifier: Operating System :: OS Independent
|
14
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
15
|
+
Requires-Python: >=3.10
|
16
|
+
Description-Content-Type: text/markdown
|
17
|
+
License-File: LICENSE
|
18
|
+
Requires-Dist: redis>=5.0.0
|
19
|
+
Requires-Dist: cachetools>=5.3.2
|
20
|
+
Provides-Extra: test
|
21
|
+
Requires-Dist: pytest>=7.4.3; extra == "test"
|
22
|
+
Requires-Dist: pytest-cov>=4.1.0; extra == "test"
|
23
|
+
Requires-Dist: pytest-mock>=3.12.0; extra == "test"
|
24
|
+
Requires-Dist: fakeredis[lua]>=2.20.1; extra == "test"
|
25
|
+
Requires-Dist: flake8>=6.1.0; extra == "test"
|
26
|
+
Requires-Dist: freezegun>=1.4.0; extra == "test"
|
27
|
+
Provides-Extra: docs
|
28
|
+
Requires-Dist: sphinx>=7.0.0; extra == "docs"
|
29
|
+
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "docs"
|
30
|
+
Requires-Dist: sphinx-git>=11.0.0; extra == "docs"
|
31
|
+
Requires-Dist: sphinxcontrib-mermaid>=0.7.1; extra == "docs"
|
32
|
+
Provides-Extra: dev
|
33
|
+
Requires-Dist: pytest>=7.4.3; extra == "dev"
|
34
|
+
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
|
35
|
+
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
|
36
|
+
Requires-Dist: fakeredis[lua]>=2.20.1; extra == "dev"
|
37
|
+
Requires-Dist: flake8>=6.1.0; extra == "dev"
|
38
|
+
Requires-Dist: freezegun>=1.4.0; extra == "dev"
|
39
|
+
Requires-Dist: sphinx>=7.0.0; extra == "dev"
|
40
|
+
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "dev"
|
41
|
+
Requires-Dist: sphinx-git>=11.0.0; extra == "dev"
|
42
|
+
Requires-Dist: sphinxcontrib-mermaid>=0.7.1; extra == "dev"
|
43
|
+
Dynamic: author
|
44
|
+
Dynamic: author-email
|
45
|
+
Dynamic: classifier
|
46
|
+
Dynamic: description
|
47
|
+
Dynamic: description-content-type
|
48
|
+
Dynamic: home-page
|
49
|
+
Dynamic: license
|
50
|
+
Dynamic: license-file
|
51
|
+
Dynamic: provides-extra
|
52
|
+
Dynamic: requires-dist
|
53
|
+
Dynamic: requires-python
|
54
|
+
Dynamic: summary
|
55
|
+
|
56
|
+
# RedisAllocator
|
57
|
+
|
58
|
+
## Project Overview
|
59
|
+
|
60
|
+
RedisAllocator provides robust and efficient components for managing distributed resources using Redis. It's designed specifically for scenarios requiring high availability, automatic recovery, and flexible allocation strategies, such as managing pools of proxies, workers, or other limited resources.
|
61
|
+
|
62
|
+
The core philosophy is to leverage Redis's speed and atomic Lua scripting capabilities to ensure consistency and performance for resource allocation, locking, and task queuing, while operating within a single Redis instance for simplicity and atomicity guarantees.
|
63
|
+
|
64
|
+
> **Note**: RedisAllocator is optimized for single Redis instance deployments. Its reliance on Lua scripting for atomicity makes it unsuitable for standard Redis Cluster configurations. For cluster environments, consider alternative locking mechanisms like RedLock.
|
65
|
+
|
66
|
+
## Core Design Principles & Features
|
67
|
+
|
68
|
+
RedisAllocator is built around these key ideas:
|
69
|
+
|
70
|
+
- **Efficient Resource Pooling:** Manages a pool of available resources, enabling clients to check out (allocate) and return (free) resources.
|
71
|
+
- **Atomic Operations:** Utilizes Redis Lua scripts extensively to guarantee atomicity for critical pool management operations, preventing race conditions in distributed environments.
|
72
|
+
- **Automatic Recovery (Garbage Collection):** Implements configurable garbage collection to automatically detect and recycle resources that are no longer in use (e.g., due to client crashes or expired locks), crucial for maintaining pool health.
|
73
|
+
- **Flexible Allocation Modes:** Supports both **exclusive** (`shared=False`, default) allocation where a resource is locked for one client, and **shared** (`shared=True`) allocation where multiple clients can use the same resource concurrently.
|
74
|
+
- **Resource Prioritization (Planned):** A key upcoming feature allowing resources to be allocated based on defined priorities (e.g., allocating faster proxies first), using Redis Sorted Sets.
|
75
|
+
- **Resource Affinity (Soft Binding):** Allows associating specific names (e.g., a worker ID or a specific task type) with resources, enabling consistent reuse of the same resource for that name, useful for caching or specialized tasks.
|
76
|
+
- **Distributed Locking:** Provides a standalone, robust distributed lock (`RedisLock`) with automatic expiry and reentrancy support.
|
77
|
+
- **Task Queuing:** Includes a basic distributed task queue (`RedisTaskQueue`) for coordinating work among multiple consumers.
|
78
|
+
|
79
|
+
### Core Features
|
80
|
+
|
81
|
+
- **Distributed Locking**: Provides robust distributed locking mechanisms to ensure data consistency in concurrent environments
|
82
|
+
- **Resource Allocation**: Implements a distributed resource allocation system with support for:
|
83
|
+
- Priority-based distribution
|
84
|
+
- Soft binding
|
85
|
+
- Garbage collection
|
86
|
+
- Health checking
|
87
|
+
- **Task Management**: Implements a distributed task queue system for efficient task processing across multiple workers
|
88
|
+
- **Object Allocation**: Supports allocation of resources with priority-based distribution and soft binding
|
89
|
+
- **Health Checking**: Monitors the health of distributed instances and automatically handles unhealthy resources
|
90
|
+
- **Garbage Collection**: Automatically identifies and reclaims unused resources, optimizing memory usage
|
91
|
+
- **Shared Mode**: Configurable allocation modes supporting both exclusive and shared resource usage
|
92
|
+
- **Soft Binding**: Associates named objects with specific resources for consistent allocation
|
93
|
+
|
94
|
+
## Documentation
|
95
|
+
|
96
|
+
For complete documentation, please visit our [official documentation site](https://invoker-bot.github.io/RedisAllocator-python/).
|
97
|
+
|
98
|
+
## Installation
|
99
|
+
|
100
|
+
```bash
|
101
|
+
pip install redis-allocator
|
102
|
+
```
|
103
|
+
|
104
|
+
## Quick Start
|
105
|
+
|
106
|
+
### Using RedisLock for Distributed Locking
|
107
|
+
|
108
|
+
RedisLock provides distributed locking with the following important characteristics:
|
109
|
+
|
110
|
+
- **Automatic Expiry**: Locks are automatically released after a timeout period, preventing deadlocks when clients fail
|
111
|
+
- **Active Update Required**: Lock holders must actively update their locks to maintain ownership
|
112
|
+
- **Thread Identification**: Each lock can include a thread identifier to determine ownership
|
113
|
+
- **Reentrant Locking**: Same thread/process can reacquire its own locks using the rlock method
|
114
|
+
|
115
|
+
```python
|
116
|
+
from redis import Redis
|
117
|
+
from redis_allocator import RedisLock
|
118
|
+
import threading
|
119
|
+
import time
|
120
|
+
|
121
|
+
# Initialize Redis client (requires a single Redis instance)
|
122
|
+
redis = Redis(host='localhost', port=6379, decode_responses=True)
|
123
|
+
|
124
|
+
# Create a RedisLock instance
|
125
|
+
lock = RedisLock(redis, "myapp", "resource-lock")
|
126
|
+
|
127
|
+
# Use the current thread ID as the lock identifier
|
128
|
+
thread_id = str(threading.get_ident())
|
129
|
+
|
130
|
+
# Acquire a lock with a 60-second timeout
|
131
|
+
if lock.lock("resource-123", value=thread_id, timeout=60):
|
132
|
+
try:
|
133
|
+
# Perform operations with the locked resource
|
134
|
+
print("Resource locked successfully")
|
135
|
+
|
136
|
+
# For long-running operations, periodically update the lock
|
137
|
+
# to prevent timeout expiration
|
138
|
+
for _ in range(5):
|
139
|
+
time.sleep(10) # Do some work
|
140
|
+
|
141
|
+
# Extend the lock's lifetime by updating it
|
142
|
+
lock.update("resource-123", value=thread_id, timeout=60)
|
143
|
+
print("Lock updated, timeout extended")
|
144
|
+
|
145
|
+
# Example of reentrant locking with rlock (succeeds because same thread_id)
|
146
|
+
if lock.rlock("resource-123", value=thread_id):
|
147
|
+
print("Successfully re-locked the resource")
|
148
|
+
finally:
|
149
|
+
# Release the lock when done
|
150
|
+
lock.unlock("resource-123")
|
151
|
+
print("Resource unlocked")
|
152
|
+
else:
|
153
|
+
print("Could not acquire lock - resource is busy")
|
154
|
+
```
|
155
|
+
|
156
|
+
**Key Concepts:**
|
157
|
+
- If a lock holder fails to update within the timeout period, the lock is automatically released
|
158
|
+
- Using `rlock()` allows the same thread to reacquire a lock it already holds
|
159
|
+
- This implementation only works with a single Redis instance (not Redis Cluster)
|
160
|
+
- In a distributed system, each node should use a unique identifier as the lock value
|
161
|
+
|
162
|
+
**Simplified Lock Flow:**
|
163
|
+
|
164
|
+
```mermaid
|
165
|
+
sequenceDiagram
|
166
|
+
participant Client
|
167
|
+
participant RedisLock
|
168
|
+
participant Redis
|
169
|
+
|
170
|
+
Client->>RedisLock: lock("key", "id", timeout=60)
|
171
|
+
RedisLock->>Redis: SET key id NX EX 60
|
172
|
+
alt Lock Acquired
|
173
|
+
Redis-->>RedisLock: OK
|
174
|
+
RedisLock-->>Client: True
|
175
|
+
Client->>RedisLock: update("key", "id", timeout=60)
|
176
|
+
RedisLock->>Redis: SET key id EX 60
|
177
|
+
Redis-->>RedisLock: OK
|
178
|
+
Client->>RedisLock: unlock("key")
|
179
|
+
RedisLock->>Redis: DEL key
|
180
|
+
Redis-->>RedisLock: 1 (deleted)
|
181
|
+
RedisLock-->>Client: True
|
182
|
+
else Lock Not Acquired (Already Locked)
|
183
|
+
Redis-->>RedisLock: nil
|
184
|
+
RedisLock-->>Client: False
|
185
|
+
end
|
186
|
+
```
|
187
|
+
|
188
|
+
### Using RedisAllocator for Resource Management
|
189
|
+
|
190
|
+
```python
|
191
|
+
from redis import Redis
|
192
|
+
from redis_allocator import RedisAllocator
|
193
|
+
|
194
|
+
# Initialize Redis client
|
195
|
+
redis = Redis(host='localhost', port=6379)
|
196
|
+
|
197
|
+
# Create a RedisAllocator instance
|
198
|
+
allocator = RedisAllocator(
|
199
|
+
redis,
|
200
|
+
prefix='myapp',
|
201
|
+
suffix='allocator',
|
202
|
+
shared=False # Whether resources can be shared
|
203
|
+
)
|
204
|
+
|
205
|
+
# Add resources to the pool
|
206
|
+
allocator.extend(['resource-1', 'resource-2', 'resource-3'])
|
207
|
+
|
208
|
+
# Allocate a resource key (returns only the key)
|
209
|
+
key = allocator.malloc_key(timeout=120)
|
210
|
+
if key:
|
211
|
+
try:
|
212
|
+
# Use the allocated resource
|
213
|
+
print(f"Allocated resource: {key}")
|
214
|
+
finally:
|
215
|
+
# Free the resource when done
|
216
|
+
allocator.free_keys(key)
|
217
|
+
|
218
|
+
# Allocate a resource with object (returns a RedisAllocatorObject)
|
219
|
+
allocated_obj = allocator.malloc(timeout=120)
|
220
|
+
if allocated_obj:
|
221
|
+
try:
|
222
|
+
# The key is available as a property
|
223
|
+
print(f"Allocated resource: {allocated_obj.key}")
|
224
|
+
|
225
|
+
# Update the resource's lock timeout
|
226
|
+
allocated_obj.update(timeout=60)
|
227
|
+
finally:
|
228
|
+
# Free the resource when done
|
229
|
+
allocator.free(allocated_obj)
|
230
|
+
|
231
|
+
# Using soft binding (associates a name with a resource)
|
232
|
+
allocator.update_soft_bind("worker-1", "resource-1")
|
233
|
+
# Later...
|
234
|
+
allocator.unbind_soft_bind("worker-1")
|
235
|
+
|
236
|
+
# Garbage collection (reclaims unused resources)
|
237
|
+
allocator.gc(count=10) # Check 10 items for cleanup
|
238
|
+
```
|
239
|
+
|
240
|
+
### Shared Mode vs Non-Shared Mode
|
241
|
+
|
242
|
+
RedisAllocator supports two allocation modes:
|
243
|
+
|
244
|
+
#### Non-shared Mode (default, `shared=False`)
|
245
|
+
- Resources are allocated exclusively to one client/thread
|
246
|
+
- When allocated, the resource is locked, preventing others from using it
|
247
|
+
- The resource remains locked until explicitly freed or until its timeout expires
|
248
|
+
- Ideal for scenarios where resources must be used exclusively
|
249
|
+
|
250
|
+
```python
|
251
|
+
# Non-shared allocator (exclusive resource usage)
|
252
|
+
exclusive_allocator = RedisAllocator(redis, "myapp", shared=False)
|
253
|
+
|
254
|
+
# When a resource is allocated, it's locked and cannot be allocated by others
|
255
|
+
key = exclusive_allocator.malloc_key(timeout=120)
|
256
|
+
if key:
|
257
|
+
# Only this client can use the key until it's freed or timeout expires
|
258
|
+
exclusive_allocator.free_keys(key)
|
259
|
+
```
|
260
|
+
|
261
|
+
#### Shared Mode (`shared=True`)
|
262
|
+
- Resources can be used concurrently by multiple clients/threads
|
263
|
+
- When allocated, the resource is made available from the free list but not locked
|
264
|
+
- Multiple clients can allocate and use the same resource simultaneously
|
265
|
+
- Ideal for read-only resources or resources that support concurrent access
|
266
|
+
|
267
|
+
```python
|
268
|
+
# Shared allocator (concurrent resource usage)
|
269
|
+
shared_allocator = RedisAllocator(redis, "myapp", shared=True)
|
270
|
+
|
271
|
+
# Resources can be accessed by multiple clients simultaneously
|
272
|
+
key = shared_allocator.malloc_key(timeout=120)
|
273
|
+
if key:
|
274
|
+
# Other clients can also allocate and use this same key
|
275
|
+
shared_allocator.free_keys(key)
|
276
|
+
```
|
277
|
+
|
278
|
+
### Soft Binding Mechanism
|
279
|
+
|
280
|
+
Soft binding creates persistent associations between named objects and allocated resources:
|
281
|
+
|
282
|
+
**Allocator Pool Structure (Conceptual):**
|
283
|
+
|
284
|
+
```mermaid
|
285
|
+
graph TD
|
286
|
+
subgraph Redis Keys
|
287
|
+
HKey["<prefix>|<suffix>|pool|head"] --> Key1["Key1: ""||Key2||Expiry"]
|
288
|
+
TKey["<prefix>|<suffix>|pool|tail"] --> KeyN["KeyN: KeyN-1||""||Expiry"]
|
289
|
+
PoolHash["<prefix>|<suffix>|pool (Hash)"]
|
290
|
+
end
|
291
|
+
|
292
|
+
subgraph "PoolHash Contents (Doubly-Linked Free List)"
|
293
|
+
Key1 --> Key2["Key2: Key1||Key3||Expiry"]
|
294
|
+
Key2 --> Key3["Key3: Key2||...||Expiry"]
|
295
|
+
Key3 --> ...
|
296
|
+
KeyN_1[...] --> KeyN
|
297
|
+
end
|
298
|
+
|
299
|
+
subgraph "Allocated Keys (Non-Shared Mode)"
|
300
|
+
LKey1["<prefix>|<suffix>:AllocatedKey1"]
|
301
|
+
LKeyX["<prefix>|<suffix>:AllocatedKeyX"]
|
302
|
+
end
|
303
|
+
|
304
|
+
subgraph "Soft Bind Cache"
|
305
|
+
CacheKey1["<prefix>|<suffix>-cache:bind:name1"] --> AllocatedKey1
|
306
|
+
end
|
307
|
+
```
|
308
|
+
|
309
|
+
**Simplified Allocation Flow (Non-Shared Mode):**
|
310
|
+
|
311
|
+
```mermaid
|
312
|
+
flowchart TD
|
313
|
+
Start --> CheckSoftBind{Soft Bind Name Provided?}
|
314
|
+
CheckSoftBind -- Yes --> GetBind{GET bind cache key}
|
315
|
+
GetBind --> IsBoundKeyValid{"Cached Key Found and Unlocked?"}
|
316
|
+
IsBoundKeyValid -- Yes --> ReturnCached[Return Cached Key]
|
317
|
+
IsBoundKeyValid -- No --> PopHead{Pop Head from Free List}
|
318
|
+
CheckSoftBind -- No --> PopHead
|
319
|
+
PopHead --> IsKeyFound{Key Found?}
|
320
|
+
IsKeyFound -- Yes --> LockKey[SET Lock Key w/ Timeout]
|
321
|
+
LockKey --> UpdateCache{"Update Bind Cache (if name provided)"}
|
322
|
+
UpdateCache --> ReturnNewKey[Return New Key]
|
323
|
+
IsKeyFound -- No --> ReturnNone[Return None]
|
324
|
+
ReturnCached --> End
|
325
|
+
ReturnNewKey --> End
|
326
|
+
ReturnNone --> End
|
327
|
+
```
|
328
|
+
|
329
|
+
**Simplified Free Flow (Non-Shared Mode):**
|
330
|
+
|
331
|
+
```mermaid
|
332
|
+
flowchart TD
|
333
|
+
Start --> DeleteLock{DEL Lock Key}
|
334
|
+
DeleteLock --> Deleted{"Key Existed? (DEL > 0)"}
|
335
|
+
Deleted -- Yes --> PushTail[Push Key to Free List Tail]
|
336
|
+
PushTail --> End
|
337
|
+
Deleted -- No --> End
|
338
|
+
```
|
339
|
+
|
340
|
+
```python
|
341
|
+
from redis import Redis
|
342
|
+
from redis_allocator import RedisAllocator, RedisAllocatableClass
|
343
|
+
|
344
|
+
# Create a custom allocatable class with a name
|
345
|
+
class MyResource(RedisAllocatableClass):
|
346
|
+
def __init__(self, resource_name):
|
347
|
+
self._name = resource_name
|
348
|
+
|
349
|
+
def set_config(self, key, params):
|
350
|
+
# Configure the resource when allocated
|
351
|
+
self.key = key
|
352
|
+
self.config = params
|
353
|
+
|
354
|
+
@property
|
355
|
+
def name(self):
|
356
|
+
# Name used for soft binding
|
357
|
+
return self._name
|
358
|
+
|
359
|
+
# Initialize allocator
|
360
|
+
redis = Redis(host='localhost', port=6379)
|
361
|
+
allocator = RedisAllocator(redis, "myapp")
|
362
|
+
|
363
|
+
# Add resources to the pool
|
364
|
+
allocator.extend(['resource-1', 'resource-2', 'resource-3'])
|
365
|
+
|
366
|
+
# Create a named resource object
|
367
|
+
resource = MyResource("database-connection")
|
368
|
+
|
369
|
+
# First allocation will assign a key from the pool
|
370
|
+
allocation1 = allocator.malloc(timeout=60, obj=resource)
|
371
|
+
print(f"First allocation: {allocation1.key}") # e.g., "resource-1"
|
372
|
+
|
373
|
+
# Free the resource
|
374
|
+
allocator.free(allocation1)
|
375
|
+
|
376
|
+
# Later allocation of the same named object will try to reuse the same key
|
377
|
+
# Can specify a custom cache timeout for the binding
|
378
|
+
allocation2 = allocator.malloc(timeout=60, obj=resource, cache_timeout=300)
|
379
|
+
print(f"Second allocation: {allocation2.key}") # Will be "resource-1" again
|
380
|
+
|
381
|
+
# Benefits of soft binding:
|
382
|
+
# 1. Resource affinity - same object gets same resource consistently
|
383
|
+
# 2. Optimized caching and resource reuse
|
384
|
+
# 3. Predictable resource mapping for debugging
|
385
|
+
```
|
386
|
+
|
387
|
+
Key features of soft binding:
|
388
|
+
- Bindings persist even after the resource is freed, with a configurable timeout
|
389
|
+
- If a bound resource is no longer available, a new resource is automatically allocated
|
390
|
+
- Explicit unbinding is available with `unbind_soft_bind(name)`
|
391
|
+
- Soft bindings have their own timeout (default 3600 seconds) separate from resource locks
|
392
|
+
|
393
|
+
### Using RedisTaskQueue for Distributed Task Processing
|
394
|
+
|
395
|
+
**Simplified Task Queue Flow:**
|
396
|
+
|
397
|
+
```mermaid
|
398
|
+
sequenceDiagram
|
399
|
+
participant Client
|
400
|
+
participant TaskQueue
|
401
|
+
participant Redis
|
402
|
+
participant Listener
|
403
|
+
|
404
|
+
Client->>TaskQueue: query(id, name, params)
|
405
|
+
TaskQueue->>Redis: SETEX result:<id> pickled_task
|
406
|
+
TaskQueue->>Redis: RPUSH queue:<name> <id>
|
407
|
+
Redis-->>TaskQueue: OK
|
408
|
+
TaskQueue-->>Client: (Waits or returns if local)
|
409
|
+
|
410
|
+
Listener->>TaskQueue: listen([name])
|
411
|
+
loop Poll Queue
|
412
|
+
TaskQueue->>Redis: BLPOP queue:<name> timeout
|
413
|
+
alt Task Available
|
414
|
+
Redis-->>TaskQueue: [queue:<name>, <id>]
|
415
|
+
TaskQueue->>Redis: GET result:<id>
|
416
|
+
Redis-->>TaskQueue: pickled_task
|
417
|
+
TaskQueue->>Listener: Execute task_fn(task)
|
418
|
+
Listener-->>TaskQueue: result/error
|
419
|
+
TaskQueue->>Redis: SETEX result:<id> updated_pickled_task
|
420
|
+
else Timeout
|
421
|
+
Redis-->>TaskQueue: nil
|
422
|
+
end
|
423
|
+
end
|
424
|
+
|
425
|
+
Client->>TaskQueue: get_task(id) (Periodically or when notified)
|
426
|
+
TaskQueue->>Redis: GET result:<id>
|
427
|
+
Redis-->>TaskQueue: updated_pickled_task
|
428
|
+
TaskQueue-->>Client: Task result/error
|
429
|
+
```
|
430
|
+
|
431
|
+
```python
|
432
|
+
from redis import Redis
|
433
|
+
from redis_allocator import RedisTaskQueue, TaskExecutePolicy
|
434
|
+
import json
|
435
|
+
|
436
|
+
# Initialize Redis client
|
437
|
+
redis = Redis(host='localhost', port=6379)
|
438
|
+
|
439
|
+
# Process tasks in a worker
|
440
|
+
def process_task(task):
|
441
|
+
# Process the task (task is a RedisTask object)
|
442
|
+
# You can access task.id, task.name, task.params
|
443
|
+
# You can update progress with task.update(current, total)
|
444
|
+
return json.dumps({"result": "processed"})
|
445
|
+
|
446
|
+
|
447
|
+
# Create a task queue
|
448
|
+
task_queue = RedisTaskQueue(redis, "myapp", task_fn=process_task)
|
449
|
+
|
450
|
+
# Submit a task with query method
|
451
|
+
result = task_queue.query(
|
452
|
+
id="task-123",
|
453
|
+
name="example-task",
|
454
|
+
params={"input": "data"},
|
455
|
+
timeout=300, # Optional timeout in seconds
|
456
|
+
policy=TaskExecutePolicy.Auto, # Execution policy
|
457
|
+
once=False # Whether to delete the result after getting it
|
458
|
+
)
|
459
|
+
|
460
|
+
# Start listening for tasks
|
461
|
+
task_queue.listen(
|
462
|
+
names=["example-task"], # List of task names to listen for
|
463
|
+
workers=128, # Number of worker threads
|
464
|
+
event=None # Optional event to signal when to stop listening
|
465
|
+
)
|
466
|
+
```
|
467
|
+
|
468
|
+
## Modules
|
469
|
+
|
470
|
+
RedisAllocator consists of several modules, each providing specific functionality:
|
471
|
+
|
472
|
+
- **lock.py**: Provides `RedisLock` and `RedisLockPool` for distributed locking mechanisms
|
473
|
+
- **task_queue.py**: Implements `RedisTaskQueue` for distributed task processing
|
474
|
+
- **allocator.py**: Contains `RedisAllocator` and related classes for resource allocation
|
475
|
+
|
476
|
+
*(Note: Internal comments and Lua script explanations within these modules have been recently refactored for better clarity.)*
|
477
|
+
|
478
|
+
### RedisAllocator Architecture
|
479
|
+
|
480
|
+
The RedisAllocator maintains resources in a doubly-linked list structure stored in Redis:
|
481
|
+
- Available resources are kept in a "free list"
|
482
|
+
- In non-shared mode, allocated resources are removed from the free list and locked
|
483
|
+
- In shared mode, allocated resources are still available for allocation by others
|
484
|
+
- The Garbage Collector periodically:
|
485
|
+
- Reclaims locked resources whose locks have expired
|
486
|
+
- Removes expired resources based on their configured timeouts
|
487
|
+
- Cleans up inconsistent states between allocations and locks
|
488
|
+
- Soft bindings are implemented as separate locks with their own timeout period
|
489
|
+
|
490
|
+
## Roadmap
|
491
|
+
|
492
|
+
* **Core Focus & Recently Completed:**
|
493
|
+
* [x] Distributed Lock (`RedisLock`, `RedisLockPool`)
|
494
|
+
* [x] Resource Allocator (`RedisAllocator`) - Exclusive & Shared modes.
|
495
|
+
* [x] Task Queue (`RedisTaskQueue`)
|
496
|
+
* [x] Soft Binding Mechanism
|
497
|
+
* [x] Basic Garbage Collection & Health Checking Foundation
|
498
|
+
* [x] Documentation Improvements (Shared Mode, Soft Binding, Lua Clarity)
|
499
|
+
* [x] Foundational Unit Tests & Allocation Mode Coverage
|
500
|
+
|
501
|
+
* **Current Development Priorities (Focusing on Proxy Pool Needs):**
|
502
|
+
* [ ] **Resource Prioritization:** Implement priority-based allocation in `RedisAllocator`, likely using Redis Sorted Sets (`ZSET`) for the free pool. *(New - High Priority)*
|
503
|
+
* [ ] **Enhanced GC & Health Checking:** Improve configurability (triggers, timeouts) and potentially add hooks for custom health validation logic. Make GC more robust for scenarios like proxy failures. *(Enhanced - High Priority)*
|
504
|
+
* [ ] **Performance Benchmarking & Optimization:** Profile core allocation, GC, and locking operations under simulated proxy pool load. Optimize Lua scripts and Python code. *(Existing - Medium Priority, relevant for performance)*
|
505
|
+
* [ ] **Enhanced Observability:** Add metrics for allocation rates, pool size, GC activity, lock contention, and soft binding usage. Improve logging. *(Existing - Medium Priority, crucial for monitoring)*
|
506
|
+
|
507
|
+
* **Future Enhancements (Single-Instance Focus):**
|
508
|
+
* **Soft Binding Helpers:** Add API methods like `get_bindings_for_key` to easily manage the proxy-to-item relationships used in the fast update mode. *(New - Medium Priority)*
|
509
|
+
* **Refined Error Handling & Recovery:** Define specific exceptions and improve robustness against Redis issues or inconsistent states. *(Existing - Medium Priority)*
|
510
|
+
* **Task Queue Improvements:** Consider task prioritization, retries, delayed tasks, batch processing, dead-letter queues (Lower priority relative to core allocator needs for now).
|
511
|
+
* **Advanced Allocator Features:** Fairness algorithms, resource weighting, custom metadata storage (Lower priority).
|
512
|
+
* **Locking Enhancements:** Contention diagnostics, fairness options (Lower priority).
|
513
|
+
* **Developer Experience:** Enhanced debugging, more complex examples (like a simplified proxy manager pattern).
|
514
|
+
|
515
|
+
* **Out of Scope (Current Direction):**
|
516
|
+
* Native Redis Cluster support.
|
517
|
+
* Multi-key atomic operations beyond single-instance Lua capabilities.
|
518
|
+
|
519
|
+
## Contributing
|
520
|
+
|
521
|
+
Contributions and suggestions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for more information.
|
522
|
+
|
523
|
+
## License
|
524
|
+
|
525
|
+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
526
|
+
|
527
|
+
## Contact
|
528
|
+
|
529
|
+
For questions or suggestions, please contact us through GitHub Issues.
|
@@ -0,0 +1,15 @@
|
|
1
|
+
redis_allocator/__init__.py,sha256=TVjUm-8YEu_MQD_PkfeIKiVknpCJBrUY9cWN1LlaZcU,1016
|
2
|
+
redis_allocator/_version.py,sha256=TZkGuMIRSRmUY3XCIs5owt2o60vXyqYMHWIkhx65uYE,22
|
3
|
+
redis_allocator/allocator.py,sha256=5JXqjH2PS5Pi_DJF9iq8b8AMWYQVSJp6SYsm4-eZ0s0,47033
|
4
|
+
redis_allocator/lock.py,sha256=fqf6WUWHKYenEArWopMIF6kWEnDfADC-bZvnQImsQVo,27400
|
5
|
+
redis_allocator/task_queue.py,sha256=8DjNr2uxhzCsHatV_CHOeGh7_K9pqQZFApSbe2blRO0,14989
|
6
|
+
redis_allocator-0.3.1.dist-info/licenses/LICENSE,sha256=Wt4X1rHpffQfEiyWcDUx8BMLjXxfPqaiYZ7Lgsj7L4c,1068
|
7
|
+
tests/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
8
|
+
tests/conftest.py,sha256=cXvJZDTUqWKaxAZfE7GWUk_cixTpMRKJd420j3vZSVs,4191
|
9
|
+
tests/test_allocator.py,sha256=hFKgLe_yONtEjjm6zssUnhK0tzihG_1xZMziztHmqqA,22404
|
10
|
+
tests/test_lock.py,sha256=MDMRNN46VhWqkHUIhYOMEDgZkFFCW_WjwRLTOjkFF-Q,46952
|
11
|
+
tests/test_task_queue.py,sha256=Fh5naikFajfOvL6GngEy_TPfOYCYZolZfVwtR6T4dTY,31710
|
12
|
+
redis_allocator-0.3.1.dist-info/METADATA,sha256=AGGUZXrJ8PPTrU2PftdpHHv8M1vCF1GQbstKD7ozcmo,21653
|
13
|
+
redis_allocator-0.3.1.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
|
14
|
+
redis_allocator-0.3.1.dist-info/top_level.txt,sha256=0hXzU7sK5FCeSolTEYxThOt3HOybnwaXv1FLRJvHVgI,22
|
15
|
+
redis_allocator-0.3.1.dist-info/RECORD,,
|
@@ -1,21 +1,21 @@
|
|
1
|
-
MIT License
|
2
|
-
|
3
|
-
Copyright (c) 2025 Invoker Bot
|
4
|
-
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
7
|
-
in the Software without restriction, including without limitation the rights
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
10
|
-
furnished to do so, subject to the following conditions:
|
11
|
-
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
13
|
-
copies or substantial portions of the Software.
|
14
|
-
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
-
SOFTWARE.
|
1
|
+
MIT License
|
2
|
+
|
3
|
+
Copyright (c) 2025 Invoker Bot
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|