ringo-task-queue 0.1.0.dev2__py3-none-win_amd64.whl → 0.1.0.dev4__py3-none-win_amd64.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ringo_task_queue/__init__.py +4 -1
- ringo_task_queue/_proto/ringo/v1/queue_pb2.py +136 -137
- ringo_task_queue/_proto/ringo/v1/queue_pb2_grpc.py +618 -619
- ringo_task_queue/_version.py +5 -3
- ringo_task_queue/bin/manifest.json +6 -6
- ringo_task_queue/bin/ringo-task-queue-windows-amd64.exe +0 -0
- ringo_task_queue/client.py +16 -3
- ringo_task_queue/concurrency.py +480 -0
- ringo_task_queue/errors.py +8 -1
- ringo_task_queue/worker.py +351 -48
- {ringo_task_queue-0.1.0.dev2.dist-info → ringo_task_queue-0.1.0.dev4.dist-info}/METADATA +54 -33
- ringo_task_queue-0.1.0.dev4.dist-info/RECORD +19 -0
- {ringo_task_queue-0.1.0.dev2.dist-info → ringo_task_queue-0.1.0.dev4.dist-info}/WHEEL +1 -1
- ringo_task_queue-0.1.0.dev2.dist-info/RECORD +0 -18
- {ringo_task_queue-0.1.0.dev2.dist-info → ringo_task_queue-0.1.0.dev4.dist-info}/licenses/LICENSE +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: ringo-task-queue
|
|
3
|
-
Version: 0.1.0.
|
|
3
|
+
Version: 0.1.0.dev4
|
|
4
4
|
Summary: Python asyncio SDK for Ringo Task Queue with a bundled offline Go daemon
|
|
5
5
|
Author: Ringo Task Queue contributors
|
|
6
6
|
License-Expression: MIT
|
|
@@ -19,7 +19,7 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
19
19
|
Classifier: Programming Language :: Python :: 3.13
|
|
20
20
|
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
21
21
|
Requires-Python: >=3.11
|
|
22
|
-
Requires-Dist: grpcio-health-checking
|
|
22
|
+
Requires-Dist: grpcio-health-checking!=1.82.*,<2,>=1.71.0
|
|
23
23
|
Requires-Dist: grpcio-status<2,>=1.71.0
|
|
24
24
|
Requires-Dist: grpcio<2,>=1.71.0
|
|
25
25
|
Requires-Dist: protobuf<7,>=5.29.5
|
|
@@ -264,13 +264,62 @@ await worker.run()
|
|
|
264
264
|
|
|
265
265
|
`run()` 一直运行到其他协程调用 `await worker.close()`,并且只能调用一次。常驻消费时用 `asyncio.create_task(worker.run())` 启动;已知有限批次(见下)直接 `await worker.run()`。
|
|
266
266
|
|
|
267
|
+
### 三种并发模式、共享代理池和 pause/resume
|
|
268
|
+
|
|
269
|
+
整数是固定并发,创建时至少为 1;运行后可以把目标改成 `0..2**31-1`:
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
worker = queue.worker(handle, concurrency=8)
|
|
273
|
+
run_task = asyncio.create_task(worker.run())
|
|
274
|
+
await worker.set_concurrency(0) # pause:不再派发,已有 handler 不取消
|
|
275
|
+
await worker.set_concurrency(8) # resume/resize
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
`ResourceConcurrency` 是同一进程/事件循环可由多个 queue/client 共享的总容量。容量是 admission permit,不是每个 Worker 的副本;缩容不会抢占已运行的 handler。
|
|
279
|
+
|
|
280
|
+
```python
|
|
281
|
+
from ringo_task_queue import ResourceConcurrency
|
|
282
|
+
|
|
283
|
+
proxies = ResourceConcurrency(capacity=12)
|
|
284
|
+
west = client.queue("west").worker(handle, concurrency=proxies, weight=1)
|
|
285
|
+
east = client.queue("east").worker(handle, concurrency=proxies, weight=3)
|
|
286
|
+
runs = [asyncio.create_task(w.run()) for w in (west, east)]
|
|
287
|
+
|
|
288
|
+
await proxies.set_capacity(20) # 例如健康代理数变化
|
|
289
|
+
# 共享 controller 时不能调用 west.set_concurrency(...)
|
|
290
|
+
|
|
291
|
+
await west.close()
|
|
292
|
+
await east.close()
|
|
293
|
+
await asyncio.gather(*runs)
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
持续有任务时,权重按平滑加权轮转长期接近 1:3;空闲成员的临时 probe 会在 250 ms 后撤回,因此容量会给活跃成员 work-conserving 使用。可手动暂停整组:`await proxies.set_capacity(0)`,恢复时再设置非零容量。
|
|
297
|
+
|
|
298
|
+
`AdaptiveConcurrency` 使用 AIMD:当前容量个成功 ACK(且当时已饱和)后 `+1`,只有成功 NACK 且 `RetryTaskError(congestion=True)` 才按 `*0.5` 降低,始终限制在 `minimum..maximum`;普通异常、永久失败和 lease/传输丢失不改变它:
|
|
299
|
+
|
|
300
|
+
```python
|
|
301
|
+
from ringo_task_queue import AdaptiveConcurrency, RetryTaskError
|
|
302
|
+
|
|
303
|
+
limits = AdaptiveConcurrency(initial=4, minimum=1, maximum=32)
|
|
304
|
+
|
|
305
|
+
async def handle(task, ctx):
|
|
306
|
+
try:
|
|
307
|
+
await call_upstream(task.payload)
|
|
308
|
+
except RateLimitError as exc:
|
|
309
|
+
raise RetryTaskError(str(exc), delay=2, congestion=True) from exc
|
|
310
|
+
|
|
311
|
+
worker = queue.worker(handle, concurrency=limits)
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
动态 controller 和运行中的 fixed resize 需要协议 minor 2;不改变 fixed 并发的 Worker 仍兼容旧 minor。controller 可以跨 queue/client 共享,但不能跨进程共享。
|
|
315
|
+
|
|
267
316
|
Handler 映射:
|
|
268
317
|
|
|
269
318
|
| 结果 | 动作 |
|
|
270
319
|
|---|---|
|
|
271
320
|
| 正常返回 | ACK / `SUCCEEDED` |
|
|
272
321
|
| 普通异常 | NACK / 按策略重试 |
|
|
273
|
-
| `RetryTaskError` | NACK
|
|
322
|
+
| `RetryTaskError` | NACK,可覆盖本次延迟;仅 `congestion=True` 且 NACK 被服务端接受时触发 Adaptive 的降速 |
|
|
274
323
|
| `PermanentTaskError` | Reject / `DEAD` |
|
|
275
324
|
| 取消 | 不发送终态,等待 lease 到期重投 |
|
|
276
325
|
|
|
@@ -421,7 +470,7 @@ except RingoError as exc:
|
|
|
421
470
|
|
|
422
471
|
| 项目 | 默认值或限制 |
|
|
423
472
|
|---|---|
|
|
424
|
-
| 协议 | 1.
|
|
473
|
+
| 协议 | 1.2 |
|
|
425
474
|
| schema | 3 |
|
|
426
475
|
| lease | 60 秒 |
|
|
427
476
|
| worker concurrency | 1 |
|
|
@@ -448,7 +497,7 @@ except RingoError as exc:
|
|
|
448
497
|
|
|
449
498
|
### `No matching distribution found`
|
|
450
499
|
|
|
451
|
-
确认 Python 为 3.11+,对应平台 wheel
|
|
500
|
+
确认 Python 为 3.11+,对应平台 wheel 已上传;安装开发版本时使用 `--pre`。
|
|
452
501
|
|
|
453
502
|
### 找不到 daemon
|
|
454
503
|
|
|
@@ -466,34 +515,6 @@ except RingoError as exc:
|
|
|
466
515
|
|
|
467
516
|
默认允许 handler 在 30 秒内完成。可以调整 `grace_period`,并让 handler 响应 cancellation 或 `ctx.cancelled`。
|
|
468
517
|
|
|
469
|
-
## 维护者构建平台 wheel
|
|
470
|
-
|
|
471
|
-
真实 daemon 不提交到源码树。必须先从同一源码版本生成并验证五平台 release manifest,再逐个平台构建 wheel:
|
|
472
|
-
|
|
473
|
-
```shell
|
|
474
|
-
go run ./tools/release build --out dist --version 0.1.0-dev --commit <revision> --date <RFC3339-UTC>
|
|
475
|
-
go run ./tools/release verify --manifest dist/manifest.json
|
|
476
|
-
```
|
|
477
|
-
|
|
478
|
-
然后从 `sdk/python` 运行:
|
|
479
|
-
|
|
480
|
-
```shell
|
|
481
|
-
python tools/build_platform_wheel.py \
|
|
482
|
-
--binary ../../dist/ringo-task-queue-linux-amd64 \
|
|
483
|
-
--release-manifest ../../dist/manifest.json \
|
|
484
|
-
--platform linux-amd64 \
|
|
485
|
-
--output-dir ../../package-dist/python
|
|
486
|
-
```
|
|
487
|
-
|
|
488
|
-
分别构建 `windows-amd64`、`linux-amd64`、`linux-arm64`、`macos-amd64`、`macos-arm64`。构建会记录根 manifest 的 SHA-256 来源,完成后清理暂存二进制。上传前运行:
|
|
489
|
-
|
|
490
|
-
```shell
|
|
491
|
-
python tools/verify_wheel_set.py ../../package-dist/python
|
|
492
|
-
python -m twine check ../../package-dist/python/*
|
|
493
|
-
```
|
|
494
|
-
|
|
495
|
-
没有经过完整五目标 manifest 暂存的 wheel 和 sdist 会被主动拒绝,避免发布不含 daemon 或平台集合不完整的包。
|
|
496
|
-
|
|
497
518
|
## 许可证
|
|
498
519
|
|
|
499
520
|
MIT License。
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
ringo_task_queue/__init__.py,sha256=zihlmp-o3e9VemeH0TKw_ahdmjgm-ykv-dZ1YkOODF8,1501
|
|
2
|
+
ringo_task_queue/_pb.py,sha256=gA8HMCx9ZXEMB-ca6me8CYXdRLLPzUXVIz63Q4bz75I,5683
|
|
3
|
+
ringo_task_queue/_version.py,sha256=vf_6CBVg0fZ61HTF04mxqE3ZyIrcJDIY9rgVAf9N9Qs,642
|
|
4
|
+
ringo_task_queue/binary.py,sha256=lJtFz_t5BuxDV4dMAqQDpP3mhTvJlnPBGnaiB0Z9D04,9526
|
|
5
|
+
ringo_task_queue/client.py,sha256=1SM_DSJvVlfYm7Yrqmtyo0DMvE2qRGNXptRNLuJ4OPI,36071
|
|
6
|
+
ringo_task_queue/concurrency.py,sha256=AxmtDa39xoda0TnvxNlLmZri1TTWCkS7VaRomAONwLU,19151
|
|
7
|
+
ringo_task_queue/daemon.py,sha256=YS8otZpD9hqKeASb2S8ZWLCWzQj6-FYUb-grmYSnrIs,19849
|
|
8
|
+
ringo_task_queue/errors.py,sha256=FQXkfz0SS-mFtY61b6f9HamPZUPWcuf-33AfB7B0rmo,4829
|
|
9
|
+
ringo_task_queue/models.py,sha256=MEXh2JKgCbf3vo0EmnjJ_TscwabaW5Yd9i98LSPIrnk,11691
|
|
10
|
+
ringo_task_queue/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
11
|
+
ringo_task_queue/worker.py,sha256=Bv7vknxThqWAzsLjQeJyzjn6Qiw8yKNRq6rCIgvEXjQ,53618
|
|
12
|
+
ringo_task_queue/_proto/ringo/v1/queue_pb2.py,sha256=knhuzUL_fMA12UstjDXXls0PvX9nJChdOKjYq3ro2yw,18598
|
|
13
|
+
ringo_task_queue/_proto/ringo/v1/queue_pb2_grpc.py,sha256=WHG2a2EA18tMYkC_NNFTuac8E72mKk7IaV0BIQwa2gw,27617
|
|
14
|
+
ringo_task_queue/bin/manifest.json,sha256=qENvp1ZOLJedA21x6UNIg-rFdllfLMxpz0Mg6V1bR4k,576
|
|
15
|
+
ringo_task_queue/bin/ringo-task-queue-windows-amd64.exe,sha256=Ty1b7P6jdHKt6r40QGXdumuliLeuQtc7TwFML4HYC9E,22876672
|
|
16
|
+
ringo_task_queue-0.1.0.dev4.dist-info/METADATA,sha256=ndIiIeTK48Eknt0D749Z16LkfjaFQ2Ve7OmMvbLlcUY,18204
|
|
17
|
+
ringo_task_queue-0.1.0.dev4.dist-info/WHEEL,sha256=A7YmR1UrJozxyeLCzQFwQUdR7CRUwkFKasEDmnrjVxY,94
|
|
18
|
+
ringo_task_queue-0.1.0.dev4.dist-info/licenses/LICENSE,sha256=yK3nnJz_bbOAPlcJqAooYcvecNJnBH9miT_X8Qrerv4,1086
|
|
19
|
+
ringo_task_queue-0.1.0.dev4.dist-info/RECORD,,
|
|
@@ -1,18 +0,0 @@
|
|
|
1
|
-
ringo_task_queue/__init__.py,sha256=MUeo1atWuaoNITczk6hDs4zYLI18IEpkVBHOE3wGhFc,1381
|
|
2
|
-
ringo_task_queue/_pb.py,sha256=gA8HMCx9ZXEMB-ca6me8CYXdRLLPzUXVIz63Q4bz75I,5683
|
|
3
|
-
ringo_task_queue/_version.py,sha256=ZzMLs6sHjRJhsqQlzaUywRpm7l_Fnt6prJ6UH1e1vAY,446
|
|
4
|
-
ringo_task_queue/binary.py,sha256=lJtFz_t5BuxDV4dMAqQDpP3mhTvJlnPBGnaiB0Z9D04,9526
|
|
5
|
-
ringo_task_queue/client.py,sha256=c7tCI5q2P0EQc_7RZcofHQHewBPVXGwtiLmKk_KUCsI,35380
|
|
6
|
-
ringo_task_queue/daemon.py,sha256=YS8otZpD9hqKeASb2S8ZWLCWzQj6-FYUb-grmYSnrIs,19849
|
|
7
|
-
ringo_task_queue/errors.py,sha256=UCPl1GJSlHzvGPO5WOmkYwLBqZj8Aozlz7sChAXR8pw,4719
|
|
8
|
-
ringo_task_queue/models.py,sha256=MEXh2JKgCbf3vo0EmnjJ_TscwabaW5Yd9i98LSPIrnk,11691
|
|
9
|
-
ringo_task_queue/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
10
|
-
ringo_task_queue/worker.py,sha256=lminZk8MQgOE2Gf6-8cNymaNb4iXpmtLERSaMxgMDqg,40317
|
|
11
|
-
ringo_task_queue/_proto/ringo/v1/queue_pb2.py,sha256=EyddzrHyXbndhxHm_EMlzLXFVMJGQNZEWgV665NM6IQ,18691
|
|
12
|
-
ringo_task_queue/_proto/ringo/v1/queue_pb2_grpc.py,sha256=b_F-9z6npDpHs3VdIJAhYfPKF6e3pJYSQTonripfWLA,28290
|
|
13
|
-
ringo_task_queue/bin/manifest.json,sha256=VXBi2vZJbqz6Cn-nl8Tp0DO402tHIEjwCR5qm-WlNsc,576
|
|
14
|
-
ringo_task_queue/bin/ringo-task-queue-windows-amd64.exe,sha256=rTg5fY_nD7RxL0OzLTNubruFUKVSyas3baXzeFaGVxo,22874624
|
|
15
|
-
ringo_task_queue-0.1.0.dev2.dist-info/METADATA,sha256=jrL95c9BbnhlnFYpq682KHt2DQ76xGF7nCNo1FIxXhY,16957
|
|
16
|
-
ringo_task_queue-0.1.0.dev2.dist-info/WHEEL,sha256=0nQjyt02uAQbhbYKWZzlDb_ZXUmJUwgypQJ-m16yWKo,94
|
|
17
|
-
ringo_task_queue-0.1.0.dev2.dist-info/licenses/LICENSE,sha256=yK3nnJz_bbOAPlcJqAooYcvecNJnBH9miT_X8Qrerv4,1086
|
|
18
|
-
ringo_task_queue-0.1.0.dev2.dist-info/RECORD,,
|
{ringo_task_queue-0.1.0.dev2.dist-info → ringo_task_queue-0.1.0.dev4.dist-info}/licenses/LICENSE
RENAMED
|
File without changes
|