adaptive-concurrency 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- adaptive_concurrency-0.1.0/.gitignore +8 -0
- adaptive_concurrency-0.1.0/LICENSE +202 -0
- adaptive_concurrency-0.1.0/NOTICE +4 -0
- adaptive_concurrency-0.1.0/PKG-INFO +387 -0
- adaptive_concurrency-0.1.0/README.md +359 -0
- adaptive_concurrency-0.1.0/benchmarks/__init__.py +0 -0
- adaptive_concurrency-0.1.0/benchmarks/sweep.py +72 -0
- adaptive_concurrency-0.1.0/docs/limitations.md +25 -0
- adaptive_concurrency-0.1.0/pyproject.toml +59 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/__init__.py +34 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/adaptive.py +551 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/aio.py +157 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/control.py +83 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/pool.py +227 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/py.typed +0 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/rate_limiter.py +167 -0
- adaptive_concurrency-0.1.0/src/adaptive_concurrency/usage_log.py +98 -0
- adaptive_concurrency-0.1.0/tests/__init__.py +0 -0
- adaptive_concurrency-0.1.0/tests/support.py +177 -0
- adaptive_concurrency-0.1.0/tests/test_adaptive.py +187 -0
- adaptive_concurrency-0.1.0/tests/test_aio.py +219 -0
- adaptive_concurrency-0.1.0/tests/test_control.py +52 -0
- adaptive_concurrency-0.1.0/tests/test_controller.py +71 -0
- adaptive_concurrency-0.1.0/tests/test_pool.py +71 -0
- adaptive_concurrency-0.1.0/tests/test_rate_limiter.py +118 -0
- adaptive_concurrency-0.1.0/tests/test_usage_log.py +88 -0
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
|
@@ -0,0 +1,387 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: adaptive-concurrency
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Latency-aware adaptive concurrency and batch sizing for bulk calls to slow, rate-limited APIs.
|
|
5
|
+
Project-URL: Homepage, https://github.com/aarajh/adaptive-concurrency
|
|
6
|
+
Project-URL: Repository, https://github.com/aarajh/adaptive-concurrency
|
|
7
|
+
Project-URL: Issues, https://github.com/aarajh/adaptive-concurrency/issues
|
|
8
|
+
Author-email: Aaraj <aarajh@users.noreply.github.com>
|
|
9
|
+
License-Expression: Apache-2.0
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
License-File: NOTICE
|
|
12
|
+
Keywords: adaptive,aimd,batching,concurrency,rate-limit,throttling
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Framework :: AsyncIO
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
23
|
+
Classifier: Topic :: Internet
|
|
24
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
25
|
+
Classifier: Typing :: Typed
|
|
26
|
+
Requires-Python: >=3.10
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
|
|
29
|
+
# adaptive-concurrency
|
|
30
|
+
|
|
31
|
+
[](https://github.com/aarajh/adaptive-concurrency/actions/workflows/ci.yml)
|
|
32
|
+
[](https://pypi.org/project/adaptive-concurrency/)
|
|
33
|
+
[](https://pypi.org/project/adaptive-concurrency/)
|
|
34
|
+
[](https://github.com/aarajh/adaptive-concurrency/blob/main/LICENSE)
|
|
35
|
+
|
|
36
|
+
Latency-aware adaptive concurrency and batch sizing for bulk calls to slow, rate-limited APIs.
|
|
37
|
+
|
|
38
|
+
It runs your batch callable over a list of items and, while it runs, tunes how many batches are in
|
|
39
|
+
flight and how many items go in each, from measured per-item latency and explicit failures. It
|
|
40
|
+
keeps a latency history for every (workers, batch size) pair it has tried, keeps an extra worker
|
|
41
|
+
only if it beats the best throughput measured at any setting by a margin, and takes live ceiling
|
|
42
|
+
changes through an audited SQLite file. Threads or asyncio. Zero runtime dependencies, Python
|
|
43
|
+
3.10+, typed.
|
|
44
|
+
|
|
45
|
+
It converges slowly under noise: from 3 workers to a knee at 6, the benchmark takes a median of 19
|
|
46
|
+
batches at a latency spread of σ=0.1 and 126 at σ=0.5 (`benchmarks/sweep.py`, sweep 4). For a short
|
|
47
|
+
job, or when you already know the server's limits, use `run_batches` with fixed settings instead.
|
|
48
|
+
Known limitations are in
|
|
49
|
+
[docs/limitations.md](https://github.com/aarajh/adaptive-concurrency/blob/main/docs/limitations.md).
|
|
50
|
+
|
|
51
|
+
## Install
|
|
52
|
+
|
|
53
|
+
```
|
|
54
|
+
pip install adaptive-concurrency
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Usage
|
|
58
|
+
|
|
59
|
+
You provide a callable that processes one batch and returns `(ok, items_written, error_message)`.
|
|
60
|
+
The runner owns dispatch, timing, tuning, logging, the failures CSV (`failures_csv`) and a circuit
|
|
61
|
+
breaker that stops after `max_consecutive_failures` (default 8) failed batches in a row. The caller
|
|
62
|
+
owns resumability (pass only items not yet done) and writing results inside the callable, counting
|
|
63
|
+
only what was written. Worker threads are reused for the whole run, so a `threading.local()` client
|
|
64
|
+
is created once per thread.
|
|
65
|
+
|
|
66
|
+
Every runner returns the total `items_written` summed over the calls that returned `ok=True`. It
|
|
67
|
+
is less than `len(items)` when batches failed or the circuit breaker stopped the run early.
|
|
68
|
+
|
|
69
|
+
Every example here runs against a local stand-in for a remote API, so you can paste it and run it.
|
|
70
|
+
This one gets slower once it has more than 8 calls in flight, and never returns an error:
|
|
71
|
+
|
|
72
|
+
```python
|
|
73
|
+
import logging
|
|
74
|
+
import threading
|
|
75
|
+
import time
|
|
76
|
+
|
|
77
|
+
from adaptive_concurrency import AdaptiveConfig, run_adaptive
|
|
78
|
+
|
|
79
|
+
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
|
80
|
+
|
|
81
|
+
in_flight = 0
|
|
82
|
+
lock = threading.Lock()
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
def call_api(batch):
|
|
86
|
+
"""Stand-in for a remote API: flat latency up to 8 concurrent calls, slower past that."""
|
|
87
|
+
global in_flight
|
|
88
|
+
with lock:
|
|
89
|
+
in_flight += 1
|
|
90
|
+
load = in_flight
|
|
91
|
+
try:
|
|
92
|
+
time.sleep(0.004 * len(batch) * (1.0 if load <= 8 else 1.0 + 0.4 * (load - 8)))
|
|
93
|
+
return True, len(batch), ""
|
|
94
|
+
finally:
|
|
95
|
+
with lock:
|
|
96
|
+
in_flight -= 1
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
config = AdaptiveConfig(workers=2, max_workers=32, batch_size=1)
|
|
100
|
+
written = run_adaptive(list(range(4000)), call_api, config)
|
|
101
|
+
print(f"wrote {written}")
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
The log records every probe, revert, and cut. Each line starts with the config's `label` in
|
|
105
|
+
brackets (default `batches`), so several jobs can share one log. This is an excerpt from one run:
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
[batches] tune: probe workers 2->3 -> workers=3, batch_size=1
|
|
109
|
+
...
|
|
110
|
+
[batches] tune: probe workers 10->11 -> workers=11, batch_size=1
|
|
111
|
+
[batches] tune: revert workers 11->10 (throughput 0.51x best) -> workers=10, batch_size=1
|
|
112
|
+
[batches] tune: decrease workers 10->7 (latency 1.77x this setting's history) -> workers=7, batch_size=1
|
|
113
|
+
[batches] tune: probe workers 7->8 -> workers=8, batch_size=1
|
|
114
|
+
[batches] tune: probe workers 8->9 -> workers=9, batch_size=1
|
|
115
|
+
[batches] tune: revert workers 9->8 (throughput 0.81x best) -> workers=8, batch_size=1
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
It overshoots, cuts, and then holds at 8. It keeps re-testing 9 now and then, because a real
|
|
119
|
+
server's capacity changes. The exact path varies from run to run, because the threads race.
|
|
120
|
+
`batch_size` stays fixed because `max_batch_size` defaults to the starting `batch_size`.
|
|
121
|
+
|
|
122
|
+
## Batch size
|
|
123
|
+
|
|
124
|
+
Many bulk APIs accept a list of items per request, with a fixed overhead per request. Bigger
|
|
125
|
+
batches amortise that overhead until they run into a size limit or a timeout. Give the tuner a
|
|
126
|
+
ceiling to probe up to and, if requests time out, the timeout:
|
|
127
|
+
|
|
128
|
+
```python
|
|
129
|
+
import time
|
|
130
|
+
|
|
131
|
+
from adaptive_concurrency import AdaptiveConfig, run_adaptive
|
|
132
|
+
|
|
133
|
+
|
|
134
|
+
def call_api(batch):
|
|
135
|
+
"""Stand-in: 20 ms per request plus 1 ms per item, and batches over 60 items are rejected."""
|
|
136
|
+
if len(batch) > 60:
|
|
137
|
+
return False, 0, "HTTP 413: batch too large"
|
|
138
|
+
time.sleep(0.02 + 0.001 * len(batch))
|
|
139
|
+
return True, len(batch), ""
|
|
140
|
+
|
|
141
|
+
|
|
142
|
+
config = AdaptiveConfig(
|
|
143
|
+
workers=4,
|
|
144
|
+
max_workers=4,
|
|
145
|
+
batch_size=5,
|
|
146
|
+
max_batch_size=200,
|
|
147
|
+
max_batch_seconds=30.0,
|
|
148
|
+
bisect_on_failure=True,
|
|
149
|
+
)
|
|
150
|
+
written = run_adaptive(list(range(6000)), call_api, config)
|
|
151
|
+
print(f"wrote {written}")
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
`bisect_on_failure=True` splits a failed batch in half and retries each half, recursively. One bad
|
|
155
|
+
item in a batch of 50 then costs about a dozen extra calls to isolate, rather than losing the
|
|
156
|
+
other 49. A batch-size cut happens only when the failure followed from the size. A failure traced
|
|
157
|
+
to one bad item is left to bisection, and one during a rate-limit cooldown is ignored.
|
|
158
|
+
|
|
159
|
+
## Failures
|
|
160
|
+
|
|
161
|
+
- **A failed batch** is one whose call returned `ok=False`. It is logged, written to the failures
|
|
162
|
+
CSV, counted toward the circuit breaker, and its items are dropped unless `bisect_on_failure`
|
|
163
|
+
recovers them. It is not retried as a whole.
|
|
164
|
+
- **A raised exception** is not caught. It propagates out of the runner, and nothing new is
|
|
165
|
+
dispatched. The sync runners let batches already running finish and discard their results; the
|
|
166
|
+
async runners cancel them. The written count is lost with the return value, so keep your own
|
|
167
|
+
record of what was written.
|
|
168
|
+
- **The circuit breaker** logs `SYSTEMATIC FAILURE` at `ERROR` and returns normally, with the
|
|
169
|
+
written count so far. It raises nothing, so compare the return value with `len(items)`.
|
|
170
|
+
- **Bisection re-sends items.** If a call wrote part of its batch before returning `ok=False`, the
|
|
171
|
+
halves send those items again, so writes must be idempotent (an upsert) or all-or-nothing per
|
|
172
|
+
call. `items_written` from a call that returned `ok=False` is never counted, with or without
|
|
173
|
+
bisection. A batch counts as a success if bisection wrote any of it.
|
|
174
|
+
- **The failures CSV** has the columns `batch,error,timestamp`: `describe_batch(batch)` (default:
|
|
175
|
+
the items joined by `;`), the error, which joins the errors of every failed part with ` | `, and
|
|
176
|
+
a Unix time. There is one row per batch that failed or partly failed, and the row lists the whole
|
|
177
|
+
batch, including items bisection wrote. It is a log, not a resume file: resume by passing only the
|
|
178
|
+
items your own record says are not yet written.
|
|
179
|
+
|
|
180
|
+
## Rate limits
|
|
181
|
+
|
|
182
|
+
`RateLimiter` enforces requests per second, and optionally bytes per minute and requests per
|
|
183
|
+
day. Every call goes through it, bisection retries included. When the server rejects you, call
|
|
184
|
+
`report_breach()` and every worker waits out one shared cooldown. By default the cooldown starts
|
|
185
|
+
at 60 s, doubles on each repeat up to 960 s, and resets after 300 s with no breach.
|
|
186
|
+
|
|
187
|
+
The runner does not know what a 429 is: a rejected call is a failed batch like any other. With
|
|
188
|
+
`bisect_on_failure=True` its halves wait out the cooldown and are retried, and without it the
|
|
189
|
+
batch's items are dropped and logged to the failures CSV. Either way the rejection is not taken as
|
|
190
|
+
evidence about batch size. This stand-in server allows 30 requests per second, fewer than the
|
|
191
|
+
limiter's 50, so it rejects some calls, and the cooldown is shortened so the example is quick:
|
|
192
|
+
|
|
193
|
+
```python
|
|
194
|
+
import threading
|
|
195
|
+
import time
|
|
196
|
+
from collections import deque
|
|
197
|
+
|
|
198
|
+
from adaptive_concurrency import AdaptiveConfig, RateLimiter, run_adaptive
|
|
199
|
+
|
|
200
|
+
limiter = RateLimiter(max_per_sec=50, breach_base_cooldown=0.5, breach_max_doublings=1)
|
|
201
|
+
recent = deque()
|
|
202
|
+
lock = threading.Lock()
|
|
203
|
+
rejected = 0
|
|
204
|
+
|
|
205
|
+
|
|
206
|
+
def call_api(batch):
|
|
207
|
+
"""Stand-in: rejects any request past 30 in the last second."""
|
|
208
|
+
global rejected
|
|
209
|
+
with lock:
|
|
210
|
+
now = time.monotonic()
|
|
211
|
+
while recent and now - recent[0] > 1.0:
|
|
212
|
+
recent.popleft()
|
|
213
|
+
recent.append(now)
|
|
214
|
+
over = len(recent) > 30
|
|
215
|
+
time.sleep(0.01)
|
|
216
|
+
if over:
|
|
217
|
+
rejected += 1
|
|
218
|
+
limiter.report_breach()
|
|
219
|
+
return False, 0, "HTTP 429"
|
|
220
|
+
return True, len(batch), ""
|
|
221
|
+
|
|
222
|
+
|
|
223
|
+
config = AdaptiveConfig(workers=4, max_workers=4, batch_size=10, rate_limiter=limiter, bisect_on_failure=True)
|
|
224
|
+
written = run_adaptive(list(range(1000)), call_api, config)
|
|
225
|
+
print(f"wrote {written} of 1000, {rejected} requests rejected with 429")
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Pass `state_file=Path(...)` to keep the daily count across restarts (local calendar day). If your
|
|
229
|
+
HTTP layer knows response sizes, call `limiter.record_bytes(n)` from it so the bytes-per-minute
|
|
230
|
+
cap can engage.
|
|
231
|
+
|
|
232
|
+
## Live control
|
|
233
|
+
|
|
234
|
+
Point `control_db` at a SQLite file. The running job checks it on batch completion, at most every
|
|
235
|
+
`control_poll_seconds` (default 5 s), and applies any change without restarting:
|
|
236
|
+
|
|
237
|
+
```python
|
|
238
|
+
from pathlib import Path
|
|
239
|
+
|
|
240
|
+
from adaptive_concurrency import audit_log, clear_override, current_overrides, set_override
|
|
241
|
+
|
|
242
|
+
db = Path("control.sqlite") # the same path you passed as AdaptiveConfig(control_db=...)
|
|
243
|
+
|
|
244
|
+
set_override(db, "max_workers", 16, note="off-peak: allow more")
|
|
245
|
+
set_override(db, "force_batch_size", 25)
|
|
246
|
+
print(current_overrides(db)) # {'max_workers': '16', 'force_batch_size': '25'}
|
|
247
|
+
|
|
248
|
+
clear_override(db, "force_batch_size", note="let it tune again")
|
|
249
|
+
for row in audit_log(db):
|
|
250
|
+
print(row) # (id, timestamp, key, value, note), oldest first
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
| Key | Effect |
|
|
254
|
+
|---|---|
|
|
255
|
+
| `max_workers`, `max_batch_size` | Move the ceiling the tuner probes within. |
|
|
256
|
+
| `min_workers`, `min_batch_size` | Move the floor. |
|
|
257
|
+
| `force_workers`, `force_batch_size` | Jump to a level now, clamped to the current floor and ceiling. |
|
|
258
|
+
|
|
259
|
+
Every set and every clear is an `INSERT`. Nothing is updated or deleted. The current value of a key
|
|
260
|
+
is its latest row, and a clear writes a tombstone, so the table is both the live config and a
|
|
261
|
+
complete history of who changed what, and why. Unknown keys are stored and ignored, so a typo is a
|
|
262
|
+
no-op rather than a crash. On a run started with `max_workers=1`, the worker keys are ignored and
|
|
263
|
+
the ignore is logged.
|
|
264
|
+
|
|
265
|
+
## Usage log
|
|
266
|
+
|
|
267
|
+
`UsageLog(path)` records every call, bisection retries included: the setting it ran under, item
|
|
268
|
+
count, elapsed time, and outcome (`ok`, `fail`, or `breach`). Use it to answer "how many requests
|
|
269
|
+
did we make today?" against a quota, across restarts:
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
from pathlib import Path
|
|
273
|
+
|
|
274
|
+
from adaptive_concurrency import AdaptiveConfig, UsageLog, run_adaptive
|
|
275
|
+
|
|
276
|
+
usage = UsageLog(Path("usage.sqlite"))
|
|
277
|
+
config = AdaptiveConfig(batch_size=10, usage_log=usage)
|
|
278
|
+
run_adaptive(list(range(100)), lambda batch: (True, len(batch), ""), config)
|
|
279
|
+
print(usage.requests_today(), usage.requests_in_window(3600))
|
|
280
|
+
usage.close()
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
## Fixed settings
|
|
284
|
+
|
|
285
|
+
`run_batches` has the same callable contract with no tuning: fixed `batch_size`, fixed `workers`.
|
|
286
|
+
It still gives you bisection, the failures CSV, the rate limiter, the usage log, and the circuit
|
|
287
|
+
breaker.
|
|
288
|
+
|
|
289
|
+
```python
|
|
290
|
+
from adaptive_concurrency import PoolConfig, run_batches
|
|
291
|
+
|
|
292
|
+
written = run_batches(list(range(100)), lambda batch: (True, len(batch), ""), PoolConfig(batch_size=20, workers=4))
|
|
293
|
+
print(written)
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
## Sync or async
|
|
297
|
+
|
|
298
|
+
`run_adaptive` and `run_batches` are thread-based: `workers` is the number of threads calling your
|
|
299
|
+
function; `run_batches` with `workers=1` and `run_adaptive` with `max_workers=1` run on the
|
|
300
|
+
calling thread. `arun_adaptive` and `arun_batches`
|
|
301
|
+
are `asyncio`-native: they take a coroutine with the same contract and the same config, run on the
|
|
302
|
+
event loop with no threads, and `workers` is the number of batches in flight. Both pairs share the
|
|
303
|
+
same `Controller`, so they tune identically. Use the async runners when your client is async (an
|
|
304
|
+
`httpx.AsyncClient`, `aiohttp`), and the sync ones when it blocks. With the async runners a
|
|
305
|
+
`rate_limiter` must provide `aacquire()`, and `RateLimiter` does.
|
|
306
|
+
|
|
307
|
+
```python
|
|
308
|
+
import asyncio
|
|
309
|
+
|
|
310
|
+
from adaptive_concurrency import AdaptiveConfig, PoolConfig, arun_adaptive, arun_batches
|
|
311
|
+
|
|
312
|
+
|
|
313
|
+
async def call_api(batch):
|
|
314
|
+
await asyncio.sleep(0.001 * len(batch))
|
|
315
|
+
return True, len(batch), ""
|
|
316
|
+
|
|
317
|
+
|
|
318
|
+
async def main():
|
|
319
|
+
print(await arun_adaptive(list(range(2000)), call_api, AdaptiveConfig(workers=2, max_workers=16)))
|
|
320
|
+
print(await arun_batches(list(range(100)), call_api, PoolConfig(batch_size=20, workers=4)))
|
|
321
|
+
|
|
322
|
+
|
|
323
|
+
asyncio.run(main())
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
## Configuration
|
|
327
|
+
|
|
328
|
+
`PoolConfig` configures `run_batches` and `arun_batches`. `AdaptiveConfig` extends it for the
|
|
329
|
+
adaptive runners, where `batch_size` and `workers` are the starting point.
|
|
330
|
+
|
|
331
|
+
| Field | Default | Meaning |
|
|
332
|
+
|---|---|---|
|
|
333
|
+
| `label` | `"batches"` | Log-line prefix. |
|
|
334
|
+
| `batch_size` | `50` | Items per call. |
|
|
335
|
+
| `workers` | `1` | Concurrent calls. |
|
|
336
|
+
| `max_consecutive_failures` | `8` | Failed batches in a row that stop the run. |
|
|
337
|
+
| `rate_limiter` | `None` | A `RateLimiter`, or anything with `acquire()` (`aacquire()` for async) and `is_backing_off()`. |
|
|
338
|
+
| `failures_csv` | `None` | Path for the failures CSV; `None` writes nothing. |
|
|
339
|
+
| `bisect_on_failure` | `False` | Split and retry a failed batch. |
|
|
340
|
+
| `bisect_min_size` | `1` | Smallest part bisection retries. |
|
|
341
|
+
| `usage_log` | `None` | A `UsageLog` recording every call. |
|
|
342
|
+
| `max_workers` | `None` | Worker ceiling; `None` means the starting `workers`. `1` declares the callable not thread-safe. |
|
|
343
|
+
| `min_workers` | `1` | Worker floor. |
|
|
344
|
+
| `max_batch_size` | `None` | Batch-size ceiling; `None` means the starting `batch_size`, so batch size is not tuned. |
|
|
345
|
+
| `min_batch_size` | `1` | Batch-size floor. |
|
|
346
|
+
| `batch_step` | `None` | Additive batch-size increase; `None` means `max(1, batch_size // 4)`. |
|
|
347
|
+
| `max_batch_seconds` | `None` | Request timeout; batch growth stops short of it. |
|
|
348
|
+
| `control_db` | `None` | SQLite file for live control. |
|
|
349
|
+
| `control_poll_seconds` | `5.0` | How often `control_db` is read. |
|
|
350
|
+
|
|
351
|
+
The remaining `AdaptiveConfig` fields tune the algorithm, and the defaults suit nearly every
|
|
352
|
+
workload: `batch_seconds_headroom=0.7`, `congestion_ratio=1.5`, `good_ratio=1.2`,
|
|
353
|
+
`worker_probe_min_ratio=1.05`, `batch_probe_min_ratio=0.85`, `worker_decrease=0.7`,
|
|
354
|
+
`batch_decrease=0.5`, `min_window=5`, `max_window=60`, `target_log_se=0.08`, `baseline_epochs=5`,
|
|
355
|
+
`ceiling_ttl_epochs=10`, `history_ttl_windows=20`. Each is documented where it is defined, in
|
|
356
|
+
[`adaptive.py`](https://github.com/aarajh/adaptive-concurrency/blob/main/src/adaptive_concurrency/adaptive.py).
|
|
357
|
+
|
|
358
|
+
## Logging
|
|
359
|
+
|
|
360
|
+
Everything goes to the standard `logging` logger `"adaptive_concurrency"`. The library prints
|
|
361
|
+
nothing and writes no files unless you pass it a path.
|
|
362
|
+
|
|
363
|
+
## Development
|
|
364
|
+
|
|
365
|
+
```
|
|
366
|
+
python -m unittest discover -s tests
|
|
367
|
+
PYTHONPATH=src python -m benchmarks.sweep
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
The sweep drives `Controller` directly across noise levels and thresholds and is the evidence
|
|
371
|
+
behind the defaults. The decision rules are documented on `Controller` in
|
|
372
|
+
[`adaptive.py`](https://github.com/aarajh/adaptive-concurrency/blob/main/src/adaptive_concurrency/adaptive.py).
|
|
373
|
+
|
|
374
|
+
## Related work
|
|
375
|
+
|
|
376
|
+
- [Netflix concurrency-limits](https://github.com/Netflix/concurrency-limits) is a Java library
|
|
377
|
+
that applies TCP congestion control (Vegas, Gradient2, AIMD limits) to the number of concurrent
|
|
378
|
+
requests. It limits concurrency only, and is aimed at services and RPC clients rather than bulk
|
|
379
|
+
batch jobs.
|
|
380
|
+
- [Envoy's adaptive concurrency filter](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/adaptive_concurrency_filter)
|
|
381
|
+
adjusts outstanding requests to a cluster by comparing sampled latency against a minRTT. It
|
|
382
|
+
measures the minRTT periodically by pinning concurrency low. It is a proxy filter, and it does
|
|
383
|
+
not size batches.
|
|
384
|
+
|
|
385
|
+
## License
|
|
386
|
+
|
|
387
|
+
Apache-2.0. See [LICENSE](https://github.com/aarajh/adaptive-concurrency/blob/main/LICENSE).
|