httpx-limiter 0.2.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- httpx_limiter-0.4.0/PKG-INFO +157 -0
- httpx_limiter-0.4.0/README.md +124 -0
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/pyproject.toml +46 -67
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/__init__.py +4 -0
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/_version.py +16 -3
- httpx_limiter-0.4.0/src/httpx_limiter/abstract_rate_limiter_repository.py +74 -0
- httpx_limiter-0.4.0/src/httpx_limiter/async_limiter.py +121 -0
- httpx_limiter-0.4.0/src/httpx_limiter/async_multi_rate_limited_transport.py +92 -0
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/async_rate_limited_transport.py +16 -11
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/rate.py +14 -6
- httpx_limiter-0.4.0/src/httpx_limiter/types.py +38 -0
- httpx_limiter-0.2.0/PKG-INFO +0 -73
- httpx_limiter-0.2.0/README.md +0 -41
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/.gitignore +0 -0
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/LICENSE +0 -0
- {httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/py.typed +0 -0
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: httpx-limiter
|
|
3
|
+
Version: 0.4.0
|
|
4
|
+
Summary: A lightweight package that provides rate-limited httpx transports.
|
|
5
|
+
Project-URL: Homepage, https://github.com/Midnighter/httpx-limiter
|
|
6
|
+
Project-URL: Documentation, https://httpx-limiter.readthedocs.io
|
|
7
|
+
Project-URL: Source Code, https://github.com/Midnighter/httpx-limiter
|
|
8
|
+
Project-URL: Bug Tracker, https://github.com/Midnighter/httpx-limiter/issues
|
|
9
|
+
Project-URL: Download, https://pypi.org/project/httpx-limiter/#files
|
|
10
|
+
Author-email: "Moritz E. Beber" <midnighter@posteo.net>
|
|
11
|
+
License-Expression: Apache-2.0
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Keywords: httpx,leaky bucket,limiter,rate-limit
|
|
14
|
+
Classifier: Development Status :: 2 - Pre-Alpha
|
|
15
|
+
Classifier: Environment :: Web Environment
|
|
16
|
+
Classifier: Framework :: AsyncIO
|
|
17
|
+
Classifier: Intended Audience :: Developers
|
|
18
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
19
|
+
Classifier: Natural Language :: English
|
|
20
|
+
Classifier: Operating System :: OS Independent
|
|
21
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
24
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
25
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
26
|
+
Classifier: Topic :: Internet :: WWW/HTTP
|
|
27
|
+
Classifier: Typing :: Typed
|
|
28
|
+
Requires-Python: >=3.10
|
|
29
|
+
Requires-Dist: httpx~=0.25
|
|
30
|
+
Requires-Dist: pyrate-limiter~=3.9
|
|
31
|
+
Requires-Dist: typing-extensions~=4.6; python_version < '3.11'
|
|
32
|
+
Description-Content-Type: text/markdown
|
|
33
|
+
|
|
34
|
+
# HTTPX Limiter
|
|
35
|
+
|
|
36
|
+
| | |
|
|
37
|
+
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
38
|
+
| Package | [](https://pypi.org/project/httpx-limiter/) [](https://pypi.org/project/httpx-limiter/) |
|
|
39
|
+
| Meta | [](LICENSE) [](.github/CODE_OF_CONDUCT.md) [](https://mypy-lang.org/) [](https://github.com/ambv/black) [](https://github.com/astral-sh/ruff) |
|
|
40
|
+
| Automation | [](https://github.com/Midnighter/httpx-limiter/actions/workflows/main.yml) |
|
|
41
|
+
|
|
42
|
+
_A lightweight package that provides rate-limited httpx transports._
|
|
43
|
+
|
|
44
|
+
## Installation
|
|
45
|
+
|
|
46
|
+
The package is published on [PyPI](https://pypi.org/project/httpx-limiter/).
|
|
47
|
+
Install it, for example, with
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
pip install httpx-limiter
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Tutorial
|
|
54
|
+
|
|
55
|
+
You can limit the number of requests made by an HTTPX client using the
|
|
56
|
+
transports provided in this package. That is useful in situations when you need
|
|
57
|
+
to make a large number of asynchronous requests against endpoints that implement
|
|
58
|
+
a rate limit.
|
|
59
|
+
|
|
60
|
+
### Single Rate Limit
|
|
61
|
+
|
|
62
|
+
The simplest use case is to apply a single rate limit to all requests. If you
|
|
63
|
+
want to be able to make twenty requests per second, for example, use the
|
|
64
|
+
following code:
|
|
65
|
+
|
|
66
|
+
```python
|
|
67
|
+
import httpx
|
|
68
|
+
from httpx_limiter import AsyncRateLimitedTransport, Rate
|
|
69
|
+
|
|
70
|
+
async def main():
|
|
71
|
+
async with httpx.AsyncClient(
|
|
72
|
+
transport=AsyncRateLimitedTransport.create(Rate.create(magnitude=20)),
|
|
73
|
+
) as client:
|
|
74
|
+
response = await client.get("https://httpbin.org")
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
> [!IMPORTANT]
|
|
78
|
+
> Due to limitations in the design of the underlying [leaky
|
|
79
|
+
> bucket](https://en.wikipedia.org/wiki/Leaky_bucket) implementation, which is
|
|
80
|
+
> used to implement the rate limiting, the magnitude of the rate is also the
|
|
81
|
+
> maximum capacity of the bucket. That means, if you set a rate that is larger
|
|
82
|
+
> than one, a burst of requests equal to that capacity will be allowed. If you
|
|
83
|
+
> do not want to allow any bursts, set the magnitude to one, but the duration to
|
|
84
|
+
> the inverse of your desired rate. If you want to allow twenty requests per
|
|
85
|
+
> second, for example, set the magnitude to 1 and the duration to 0.05 seconds.
|
|
86
|
+
>
|
|
87
|
+
> ```python
|
|
88
|
+
> Rate.create(magnitude=1, duration=1/20)
|
|
89
|
+
> ```
|
|
90
|
+
|
|
91
|
+
### Multiple Rate Limits
|
|
92
|
+
|
|
93
|
+
For more advanced use cases, you can apply different rate limits based on a
|
|
94
|
+
concrete implementation of the `AbstractRateLimiterRepository`. There are two
|
|
95
|
+
relevant methods that both get passed the current request. One method needs to
|
|
96
|
+
identify which rate limit to apply, and the other method sets the rate limit
|
|
97
|
+
itself. See the following example:
|
|
98
|
+
|
|
99
|
+
```python
|
|
100
|
+
import httpx
|
|
101
|
+
from httpx_limiter import (
|
|
102
|
+
AbstractRateLimiterRepository,
|
|
103
|
+
AsyncMultiRateLimitedTransport,
|
|
104
|
+
Rate
|
|
105
|
+
)
|
|
106
|
+
|
|
107
|
+
class DomainBasedRateLimiterRepository(AbstractRateLimiterRepository):
|
|
108
|
+
"""Apply different rate limits based on the domain being requested."""
|
|
109
|
+
|
|
110
|
+
def get_identifier(self, request: httpx.Request) -> str:
|
|
111
|
+
"""Return the domain as the identifier for rate limiting."""
|
|
112
|
+
return request.url.host
|
|
113
|
+
|
|
114
|
+
def get_rate(self, request: httpx.Request) -> Rate:
|
|
115
|
+
"""Apply the same, but independent rate limit to each domain."""
|
|
116
|
+
return Rate.create(magnitude=25)
|
|
117
|
+
|
|
118
|
+
client = httpx.AsyncClient(
|
|
119
|
+
transport=AsyncMultiRateLimitedTransport.create(
|
|
120
|
+
repository=DomainBasedRateLimiterRepository(),
|
|
121
|
+
),
|
|
122
|
+
)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
> [!TIP]
|
|
126
|
+
> You are free to ignore the request parameter and use global information like
|
|
127
|
+
> the time of day to determine the rate limit.
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
from datetime import datetime, timezone
|
|
131
|
+
from collections.abc import Sequence
|
|
132
|
+
|
|
133
|
+
import httpx
|
|
134
|
+
from httpx_limiter import AbstractRateLimiterRepository, Rate
|
|
135
|
+
|
|
136
|
+
class DayNightRateLimiterRepository(AbstractRateLimiterRepository):
|
|
137
|
+
"""Apply different rate limits based on the time of day."""
|
|
138
|
+
|
|
139
|
+
def get_identifier(self, _: httpx.Request) -> str:
|
|
140
|
+
"""Identify whether it is currently day or night."""
|
|
141
|
+
if 6 <= datetime.now(tz=timezone.utc).hour < 18:
|
|
142
|
+
return "day"
|
|
143
|
+
|
|
144
|
+
return "night"
|
|
145
|
+
|
|
146
|
+
def get_rates(self, _: httpx.Request) -> Sequence[Rate]:
|
|
147
|
+
"""Apply different rate limits during the day or night."""
|
|
148
|
+
if self.get_identifier(_) == "day":
|
|
149
|
+
return [Rate.create(magnitude=10)]
|
|
150
|
+
|
|
151
|
+
return [Rate.create(magnitude=100)]
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
## Copyright
|
|
155
|
+
|
|
156
|
+
- Copyright © 2024, 2025 Moritz E. Beber.
|
|
157
|
+
- Free software distributed under the [Apache Software License 2.0](./LICENSE).
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# HTTPX Limiter
|
|
2
|
+
|
|
3
|
+
| | |
|
|
4
|
+
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
5
|
+
| Package | [](https://pypi.org/project/httpx-limiter/) [](https://pypi.org/project/httpx-limiter/) |
|
|
6
|
+
| Meta | [](LICENSE) [](.github/CODE_OF_CONDUCT.md) [](https://mypy-lang.org/) [](https://github.com/ambv/black) [](https://github.com/astral-sh/ruff) |
|
|
7
|
+
| Automation | [](https://github.com/Midnighter/httpx-limiter/actions/workflows/main.yml) |
|
|
8
|
+
|
|
9
|
+
_A lightweight package that provides rate-limited httpx transports._
|
|
10
|
+
|
|
11
|
+
## Installation
|
|
12
|
+
|
|
13
|
+
The package is published on [PyPI](https://pypi.org/project/httpx-limiter/).
|
|
14
|
+
Install it, for example, with
|
|
15
|
+
|
|
16
|
+
```sh
|
|
17
|
+
pip install httpx-limiter
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Tutorial
|
|
21
|
+
|
|
22
|
+
You can limit the number of requests made by an HTTPX client using the
|
|
23
|
+
transports provided in this package. That is useful in situations when you need
|
|
24
|
+
to make a large number of asynchronous requests against endpoints that implement
|
|
25
|
+
a rate limit.
|
|
26
|
+
|
|
27
|
+
### Single Rate Limit
|
|
28
|
+
|
|
29
|
+
The simplest use case is to apply a single rate limit to all requests. If you
|
|
30
|
+
want to be able to make twenty requests per second, for example, use the
|
|
31
|
+
following code:
|
|
32
|
+
|
|
33
|
+
```python
|
|
34
|
+
import httpx
|
|
35
|
+
from httpx_limiter import AsyncRateLimitedTransport, Rate
|
|
36
|
+
|
|
37
|
+
async def main():
|
|
38
|
+
async with httpx.AsyncClient(
|
|
39
|
+
transport=AsyncRateLimitedTransport.create(Rate.create(magnitude=20)),
|
|
40
|
+
) as client:
|
|
41
|
+
response = await client.get("https://httpbin.org")
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
> [!IMPORTANT]
|
|
45
|
+
> Due to limitations in the design of the underlying [leaky
|
|
46
|
+
> bucket](https://en.wikipedia.org/wiki/Leaky_bucket) implementation, which is
|
|
47
|
+
> used to implement the rate limiting, the magnitude of the rate is also the
|
|
48
|
+
> maximum capacity of the bucket. That means, if you set a rate that is larger
|
|
49
|
+
> than one, a burst of requests equal to that capacity will be allowed. If you
|
|
50
|
+
> do not want to allow any bursts, set the magnitude to one, but the duration to
|
|
51
|
+
> the inverse of your desired rate. If you want to allow twenty requests per
|
|
52
|
+
> second, for example, set the magnitude to 1 and the duration to 0.05 seconds.
|
|
53
|
+
>
|
|
54
|
+
> ```python
|
|
55
|
+
> Rate.create(magnitude=1, duration=1/20)
|
|
56
|
+
> ```
|
|
57
|
+
|
|
58
|
+
### Multiple Rate Limits
|
|
59
|
+
|
|
60
|
+
For more advanced use cases, you can apply different rate limits based on a
|
|
61
|
+
concrete implementation of the `AbstractRateLimiterRepository`. There are two
|
|
62
|
+
relevant methods that both get passed the current request. One method needs to
|
|
63
|
+
identify which rate limit to apply, and the other method sets the rate limit
|
|
64
|
+
itself. See the following example:
|
|
65
|
+
|
|
66
|
+
```python
|
|
67
|
+
import httpx
|
|
68
|
+
from httpx_limiter import (
|
|
69
|
+
AbstractRateLimiterRepository,
|
|
70
|
+
AsyncMultiRateLimitedTransport,
|
|
71
|
+
Rate
|
|
72
|
+
)
|
|
73
|
+
|
|
74
|
+
class DomainBasedRateLimiterRepository(AbstractRateLimiterRepository):
|
|
75
|
+
"""Apply different rate limits based on the domain being requested."""
|
|
76
|
+
|
|
77
|
+
def get_identifier(self, request: httpx.Request) -> str:
|
|
78
|
+
"""Return the domain as the identifier for rate limiting."""
|
|
79
|
+
return request.url.host
|
|
80
|
+
|
|
81
|
+
def get_rate(self, request: httpx.Request) -> Rate:
|
|
82
|
+
"""Apply the same, but independent rate limit to each domain."""
|
|
83
|
+
return Rate.create(magnitude=25)
|
|
84
|
+
|
|
85
|
+
client = httpx.AsyncClient(
|
|
86
|
+
transport=AsyncMultiRateLimitedTransport.create(
|
|
87
|
+
repository=DomainBasedRateLimiterRepository(),
|
|
88
|
+
),
|
|
89
|
+
)
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
> [!TIP]
|
|
93
|
+
> You are free to ignore the request parameter and use global information like
|
|
94
|
+
> the time of day to determine the rate limit.
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
from datetime import datetime, timezone
|
|
98
|
+
from collections.abc import Sequence
|
|
99
|
+
|
|
100
|
+
import httpx
|
|
101
|
+
from httpx_limiter import AbstractRateLimiterRepository, Rate
|
|
102
|
+
|
|
103
|
+
class DayNightRateLimiterRepository(AbstractRateLimiterRepository):
|
|
104
|
+
"""Apply different rate limits based on the time of day."""
|
|
105
|
+
|
|
106
|
+
def get_identifier(self, _: httpx.Request) -> str:
|
|
107
|
+
"""Identify whether it is currently day or night."""
|
|
108
|
+
if 6 <= datetime.now(tz=timezone.utc).hour < 18:
|
|
109
|
+
return "day"
|
|
110
|
+
|
|
111
|
+
return "night"
|
|
112
|
+
|
|
113
|
+
def get_rates(self, _: httpx.Request) -> Sequence[Rate]:
|
|
114
|
+
"""Apply different rate limits during the day or night."""
|
|
115
|
+
if self.get_identifier(_) == "day":
|
|
116
|
+
return [Rate.create(magnitude=10)]
|
|
117
|
+
|
|
118
|
+
return [Rate.create(magnitude=100)]
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
## Copyright
|
|
122
|
+
|
|
123
|
+
- Copyright © 2024, 2025 Moritz E. Beber.
|
|
124
|
+
- Free software distributed under the [Apache Software License 2.0](./LICENSE).
|
|
@@ -14,11 +14,9 @@ requires = ["hatchling", "hatch-vcs"]
|
|
|
14
14
|
name = "httpx-limiter"
|
|
15
15
|
dynamic = ["version"]
|
|
16
16
|
description = "A lightweight package that provides rate-limited httpx transports."
|
|
17
|
-
authors = [
|
|
18
|
-
{ name = "Moritz E. Beber", email = "midnighter@posteo.net" },
|
|
19
|
-
]
|
|
17
|
+
authors = [{ name = "Moritz E. Beber", email = "midnighter@posteo.net" }]
|
|
20
18
|
license = "Apache-2.0"
|
|
21
|
-
readme = {"file" = "README.md", "content-type" = "text/markdown"}
|
|
19
|
+
readme = { "file" = "README.md", "content-type" = "text/markdown" }
|
|
22
20
|
requires-python = ">=3.10"
|
|
23
21
|
# Please consult https://pypi.org/classifiers/ for a full list.
|
|
24
22
|
classifiers = [
|
|
@@ -37,15 +35,11 @@ classifiers = [
|
|
|
37
35
|
"Topic :: Internet :: WWW/HTTP",
|
|
38
36
|
"Typing :: Typed",
|
|
39
37
|
]
|
|
40
|
-
keywords = [
|
|
41
|
-
"httpx",
|
|
42
|
-
"limiter",
|
|
43
|
-
"rate-limit",
|
|
44
|
-
"leaky bucket",
|
|
45
|
-
]
|
|
38
|
+
keywords = ["httpx", "limiter", "rate-limit", "leaky bucket"]
|
|
46
39
|
dependencies = [
|
|
47
40
|
"httpx ~=0.25",
|
|
48
|
-
"
|
|
41
|
+
"pyrate-limiter ~=3.9",
|
|
42
|
+
"typing-extensions ~=4.6; python_version < '3.11'",
|
|
49
43
|
]
|
|
50
44
|
|
|
51
45
|
[project.urls]
|
|
@@ -76,17 +70,12 @@ testpaths = ["tests"]
|
|
|
76
70
|
markers = ["raises"]
|
|
77
71
|
|
|
78
72
|
[tool.coverage.paths]
|
|
79
|
-
source = [
|
|
80
|
-
"src/httpx_limiter",
|
|
81
|
-
"*/site-packages/httpx_limiter",
|
|
82
|
-
]
|
|
73
|
+
source = ["src/httpx_limiter", "*/site-packages/httpx_limiter"]
|
|
83
74
|
|
|
84
75
|
[tool.coverage.run]
|
|
85
76
|
branch = true
|
|
86
77
|
parallel = true
|
|
87
|
-
omit = [
|
|
88
|
-
"src/httpx_limiter/_version.py",
|
|
89
|
-
]
|
|
78
|
+
omit = ["src/httpx_limiter/_version.py"]
|
|
90
79
|
|
|
91
80
|
[tool.coverage.report]
|
|
92
81
|
exclude_lines = ["pragma: no cover"]
|
|
@@ -98,29 +87,27 @@ line-length = 88
|
|
|
98
87
|
[tool.ruff.lint]
|
|
99
88
|
select = ["ALL"]
|
|
100
89
|
ignore = [
|
|
101
|
-
"D107",
|
|
102
|
-
"D203",
|
|
103
|
-
"D212",
|
|
104
|
-
"D407",
|
|
105
|
-
"ANN002",
|
|
106
|
-
"ANN003",
|
|
107
|
-
]
|
|
108
|
-
exclude = [
|
|
109
|
-
"src/httpx_limiter/_version.py",
|
|
90
|
+
"D107", # 'Missing docstring in __init__' ignored because pydoclint wants us to document the class instead.
|
|
91
|
+
"D203", # '1 blank line required before class docstring' ignored because we want no blank line.
|
|
92
|
+
"D212", # 'Multi-line docstring summary should start at the first line' ignored because we want the summary to start on the second line.
|
|
93
|
+
"D407", # 'Missing dashed underline after section' ignored because Google style docstrings don't underline.
|
|
94
|
+
"ANN002", # 'Missing type annotation for {*args} in method'.
|
|
95
|
+
"ANN003", # 'Missing type annotation for {*kwargs} in method'.
|
|
110
96
|
]
|
|
97
|
+
exclude = ["src/httpx_limiter/_version.py"]
|
|
111
98
|
|
|
112
99
|
[tool.ruff.lint.extend-per-file-ignores]
|
|
113
100
|
"__init__.py" = [
|
|
114
|
-
"E401",
|
|
115
|
-
"E402",
|
|
116
|
-
"F401",
|
|
117
|
-
"I001",
|
|
101
|
+
"E401", # 'Multiple imports on one line'
|
|
102
|
+
"E402", # 'Module level import not at top of file'
|
|
103
|
+
"F401", # 'Imported but unused'
|
|
104
|
+
"I001", # 'Import block is un-sorted or un-formatted' ignored because we may have to import in a particular, not-alphabetical order.
|
|
118
105
|
]
|
|
119
106
|
"tests/**/*.py" = [
|
|
120
|
-
"S101",
|
|
107
|
+
"S101", # 'Use of assert detected' ignored because we are using pytest.
|
|
121
108
|
"INP001", # 'Insecure input' ignored because we are testing.
|
|
122
109
|
"ANN201", # 'Missing type annotation for {return}' ignored because all tests return `None`.
|
|
123
|
-
"PLR2004",
|
|
110
|
+
"PLR2004", # Magic numbers are okay in tests.
|
|
124
111
|
]
|
|
125
112
|
|
|
126
113
|
[tool.ruff.lint.isort]
|
|
@@ -143,9 +130,7 @@ exclude = "_version.py"
|
|
|
143
130
|
|
|
144
131
|
# Stop mypy from complaining about missing types from imports.
|
|
145
132
|
[[tool.mypy.overrides]]
|
|
146
|
-
module = [
|
|
147
|
-
"limiter",
|
|
148
|
-
]
|
|
133
|
+
module = ["limiter"]
|
|
149
134
|
ignore_missing_imports = true
|
|
150
135
|
|
|
151
136
|
#[tool.pydantic-mypy]
|
|
@@ -161,12 +146,7 @@ ignore_missing_imports = true
|
|
|
161
146
|
installer = "uv"
|
|
162
147
|
|
|
163
148
|
[tool.hatch.envs.dev]
|
|
164
|
-
description = """
|
|
165
|
-
Development environment.
|
|
166
|
-
|
|
167
|
-
Defining a path is a workaround for PyCharm,
|
|
168
|
-
see https://youtrack.jetbrains.com/issue/PY-60410.
|
|
169
|
-
"""
|
|
149
|
+
description = """Development environment."""
|
|
170
150
|
extra-dependencies = [
|
|
171
151
|
"anyio",
|
|
172
152
|
"pytest",
|
|
@@ -174,14 +154,14 @@ extra-dependencies = [
|
|
|
174
154
|
"pytest-watcher",
|
|
175
155
|
"ruff",
|
|
176
156
|
]
|
|
177
|
-
|
|
157
|
+
|
|
158
|
+
[tool.hatch.envs.dev.env-vars]
|
|
159
|
+
PYTHONDEVMODE = "1"
|
|
160
|
+
PYTHONASYNCIODEBUG = "1"
|
|
178
161
|
|
|
179
162
|
[tool.hatch.envs.style]
|
|
180
163
|
description = """Check the style of the codebase."""
|
|
181
|
-
dependencies = [
|
|
182
|
-
"pydoclint ~=0.6",
|
|
183
|
-
"ruff",
|
|
184
|
-
]
|
|
164
|
+
dependencies = ["pydoclint", "ruff"]
|
|
185
165
|
detached = true
|
|
186
166
|
installer = "uv"
|
|
187
167
|
|
|
@@ -193,18 +173,14 @@ check = ["docstrings", "code"]
|
|
|
193
173
|
|
|
194
174
|
[tool.hatch.envs.audit]
|
|
195
175
|
description = """Check dependencies for security vulnerabilities."""
|
|
196
|
-
extra-dependencies = [
|
|
197
|
-
"pip-audit",
|
|
198
|
-
]
|
|
176
|
+
extra-dependencies = ["pip-audit"]
|
|
199
177
|
|
|
200
178
|
[tool.hatch.envs.audit.scripts]
|
|
201
179
|
check = ["pip-audit"]
|
|
202
180
|
|
|
203
181
|
[tool.hatch.envs.types]
|
|
204
182
|
description = """Check the static types of the codebase."""
|
|
205
|
-
dependencies = [
|
|
206
|
-
"mypy",
|
|
207
|
-
]
|
|
183
|
+
dependencies = ["mypy"]
|
|
208
184
|
|
|
209
185
|
[tool.hatch.envs.types.scripts]
|
|
210
186
|
check = "mypy src/httpx_limiter"
|
|
@@ -215,26 +191,22 @@ extra-dependencies = [
|
|
|
215
191
|
"mkdocs-material ~=9.5",
|
|
216
192
|
"mkdocstrings[python] ~=0.24",
|
|
217
193
|
"mkdocs-awesome-pages-plugin ~=2.9",
|
|
194
|
+
"mike ~=2.1",
|
|
218
195
|
]
|
|
219
196
|
|
|
220
197
|
[tool.hatch.envs.docs.scripts]
|
|
221
198
|
build = "mkdocs build {args:--clean --strict}"
|
|
222
199
|
serve = "mkdocs serve {args}"
|
|
200
|
+
deploy = "mike deploy {args}"
|
|
223
201
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
#]
|
|
230
|
-
#detached = true
|
|
202
|
+
[tool.hatch.envs.distribution]
|
|
203
|
+
description = """Test the source and wheel distribution of the package."""
|
|
204
|
+
dependencies = ["pip", "twine"]
|
|
205
|
+
detached = true
|
|
206
|
+
installer = "uv"
|
|
231
207
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
# "pip check",
|
|
235
|
-
# "hatch build {args:--clean}",
|
|
236
|
-
# "twine check dist/*",
|
|
237
|
-
#]
|
|
208
|
+
[tool.hatch.envs.distribution.scripts]
|
|
209
|
+
check = ["pip check", "hatch build {args:--clean}", "twine check dist/*"]
|
|
238
210
|
|
|
239
211
|
[tool.hatch.envs.unit-tests]
|
|
240
212
|
description = """Run the unit tests."""
|
|
@@ -246,6 +218,10 @@ extra-dependencies = [
|
|
|
246
218
|
"pytest-raises",
|
|
247
219
|
]
|
|
248
220
|
|
|
221
|
+
[tool.hatch.envs.unit-tests.env-vars]
|
|
222
|
+
PYTHONDEVMODE = "1"
|
|
223
|
+
PYTHONASYNCIODEBUG = "1"
|
|
224
|
+
|
|
249
225
|
[[tool.hatch.envs.unit-tests.matrix]]
|
|
250
226
|
python = ["3.10", "3.11", "3.12", "3.13"]
|
|
251
227
|
|
|
@@ -256,13 +232,16 @@ run = "pytest {args:--cov=httpx_limiter --cov-report=term-missing} {root}/tests/
|
|
|
256
232
|
description = """Run the acceptance tests."""
|
|
257
233
|
extra-dependencies = [
|
|
258
234
|
"anyio",
|
|
259
|
-
# TODO: Use behave in future.
|
|
260
235
|
"pytest",
|
|
261
236
|
"pytest-cov",
|
|
262
237
|
"pytest-httpx",
|
|
263
238
|
"pytest-raises",
|
|
264
239
|
]
|
|
265
240
|
|
|
241
|
+
[tool.hatch.envs.acceptance-tests.env-vars]
|
|
242
|
+
PYTHONDEVMODE = "1"
|
|
243
|
+
PYTHONASYNCIODEBUG = "1"
|
|
244
|
+
|
|
266
245
|
[[tool.hatch.envs.acceptance-tests.matrix]]
|
|
267
246
|
python = ["3.10", "3.11", "3.12", "3.13"]
|
|
268
247
|
|
|
@@ -15,5 +15,9 @@
|
|
|
15
15
|
|
|
16
16
|
"""Provide top level symbols."""
|
|
17
17
|
|
|
18
|
+
from .types import HTTPXAsyncHTTPTransportKeywordArguments
|
|
18
19
|
from .rate import Number, Rate
|
|
20
|
+
from .async_limiter import AsyncLimiter, PyRateLimiterKeywordArguments
|
|
19
21
|
from .async_rate_limited_transport import AsyncRateLimitedTransport
|
|
22
|
+
from .abstract_rate_limiter_repository import AbstractRateLimiterRepository
|
|
23
|
+
from .async_multi_rate_limited_transport import AsyncMultiRateLimitedTransport
|
|
@@ -1,7 +1,14 @@
|
|
|
1
1
|
# file generated by setuptools-scm
|
|
2
2
|
# don't change, don't track in version control
|
|
3
3
|
|
|
4
|
-
__all__ = [
|
|
4
|
+
__all__ = [
|
|
5
|
+
"__version__",
|
|
6
|
+
"__version_tuple__",
|
|
7
|
+
"version",
|
|
8
|
+
"version_tuple",
|
|
9
|
+
"__commit_id__",
|
|
10
|
+
"commit_id",
|
|
11
|
+
]
|
|
5
12
|
|
|
6
13
|
TYPE_CHECKING = False
|
|
7
14
|
if TYPE_CHECKING:
|
|
@@ -9,13 +16,19 @@ if TYPE_CHECKING:
|
|
|
9
16
|
from typing import Union
|
|
10
17
|
|
|
11
18
|
VERSION_TUPLE = Tuple[Union[int, str], ...]
|
|
19
|
+
COMMIT_ID = Union[str, None]
|
|
12
20
|
else:
|
|
13
21
|
VERSION_TUPLE = object
|
|
22
|
+
COMMIT_ID = object
|
|
14
23
|
|
|
15
24
|
version: str
|
|
16
25
|
__version__: str
|
|
17
26
|
__version_tuple__: VERSION_TUPLE
|
|
18
27
|
version_tuple: VERSION_TUPLE
|
|
28
|
+
commit_id: COMMIT_ID
|
|
29
|
+
__commit_id__: COMMIT_ID
|
|
19
30
|
|
|
20
|
-
__version__ = version = '0.
|
|
21
|
-
__version_tuple__ = version_tuple = (0,
|
|
31
|
+
__version__ = version = '0.4.0'
|
|
32
|
+
__version_tuple__ = version_tuple = (0, 4, 0)
|
|
33
|
+
|
|
34
|
+
__commit_id__ = commit_id = None
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Copyright (c) 2025 Moritz E. Beber
|
|
2
|
+
#
|
|
3
|
+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
|
4
|
+
# use this file except in compliance with the License. You may obtain a copy of
|
|
5
|
+
# the License at
|
|
6
|
+
#
|
|
7
|
+
# https://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
#
|
|
9
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
11
|
+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
12
|
+
# License for the specific language governing permissions and limitations under
|
|
13
|
+
# the License.
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
"""Provide an abstract repository for rate limiters."""
|
|
17
|
+
|
|
18
|
+
from abc import ABC, abstractmethod
|
|
19
|
+
from collections.abc import Sequence
|
|
20
|
+
|
|
21
|
+
import httpx
|
|
22
|
+
|
|
23
|
+
from .async_limiter import AsyncLimiter, PyRateLimiterKeywordArguments
|
|
24
|
+
from .rate import Rate
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
class AbstractRateLimiterRepository(ABC):
|
|
28
|
+
"""
|
|
29
|
+
Define the abstract repository for rate limiters.
|
|
30
|
+
|
|
31
|
+
This abstract base class provides a framework for managing rate limiters
|
|
32
|
+
based on HTTP requests. It maintains a cache of rate limiters and provides
|
|
33
|
+
methods to retrieve request-specific identifiers, rates, and limiters.
|
|
34
|
+
|
|
35
|
+
Subclasses must implement methods to determine how requests are identified
|
|
36
|
+
and what rate limits should be applied to them.
|
|
37
|
+
|
|
38
|
+
Methods:
|
|
39
|
+
get_identifier: Return a request-specific identifier.
|
|
40
|
+
get_rates: Return one or more request-specific rates.
|
|
41
|
+
get: Return a request-specific rate limiter.
|
|
42
|
+
|
|
43
|
+
"""
|
|
44
|
+
|
|
45
|
+
def __init__(self, **kwargs: dict[str, object]) -> None:
|
|
46
|
+
super().__init__(**kwargs)
|
|
47
|
+
self._limiters: dict[str, AsyncLimiter] = {}
|
|
48
|
+
|
|
49
|
+
@abstractmethod
|
|
50
|
+
def get_identifier(self, request: httpx.Request) -> str:
|
|
51
|
+
"""Return a request-specific identifier."""
|
|
52
|
+
|
|
53
|
+
@abstractmethod
|
|
54
|
+
def get_rates(self, request: httpx.Request) -> Sequence[Rate]:
|
|
55
|
+
"""Return one or more request-specific rates."""
|
|
56
|
+
|
|
57
|
+
def _get_limiter_kwargs(
|
|
58
|
+
self,
|
|
59
|
+
request: httpx.Request, # noqa: ARG002
|
|
60
|
+
) -> PyRateLimiterKeywordArguments:
|
|
61
|
+
"""Return the keyword arguments for creating a rate limiter."""
|
|
62
|
+
return {}
|
|
63
|
+
|
|
64
|
+
def get(self, request: httpx.Request) -> AsyncLimiter:
|
|
65
|
+
"""Return a request-specific rate limiter."""
|
|
66
|
+
identifier = self.get_identifier(request)
|
|
67
|
+
|
|
68
|
+
if identifier not in self._limiters:
|
|
69
|
+
self._limiters[identifier] = AsyncLimiter.create(
|
|
70
|
+
*self.get_rates(request),
|
|
71
|
+
**self._get_limiter_kwargs(request),
|
|
72
|
+
)
|
|
73
|
+
|
|
74
|
+
return self._limiters[identifier]
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# Copyright (c) 2025 Moritz E. Beber
|
|
2
|
+
#
|
|
3
|
+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
|
4
|
+
# use this file except in compliance with the License. You may obtain a copy of
|
|
5
|
+
# the License at
|
|
6
|
+
#
|
|
7
|
+
# https://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
#
|
|
9
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
11
|
+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
12
|
+
# License for the specific language governing permissions and limitations under
|
|
13
|
+
# the License.
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
"""Provide an asynchronous rate-limited transport."""
|
|
17
|
+
|
|
18
|
+
from __future__ import annotations
|
|
19
|
+
|
|
20
|
+
import sys
|
|
21
|
+
from typing import TYPE_CHECKING, TypedDict
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
if sys.version_info < (3, 11):
|
|
25
|
+
from typing_extensions import Unpack
|
|
26
|
+
else:
|
|
27
|
+
from typing import Unpack
|
|
28
|
+
|
|
29
|
+
from pyrate_limiter import (
|
|
30
|
+
AbstractClock,
|
|
31
|
+
BucketAsyncWrapper,
|
|
32
|
+
Duration,
|
|
33
|
+
InMemoryBucket,
|
|
34
|
+
Limiter,
|
|
35
|
+
TimeAsyncClock,
|
|
36
|
+
validate_rate_list,
|
|
37
|
+
)
|
|
38
|
+
from pyrate_limiter import Rate as PyRate
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
if TYPE_CHECKING: # pragma: no cover
|
|
42
|
+
from types import TracebackType
|
|
43
|
+
|
|
44
|
+
from .rate import Rate
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
class PyRateLimiterKeywordArguments(TypedDict, total=False):
|
|
48
|
+
"""Keyword arguments for the pyrate limiter."""
|
|
49
|
+
|
|
50
|
+
clock: AbstractClock
|
|
51
|
+
raise_when_fail: bool
|
|
52
|
+
max_delay: int | Duration | None
|
|
53
|
+
retry_until_max_delay: bool
|
|
54
|
+
buffer_ms: int
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
class AsyncLimiter:
|
|
58
|
+
"""
|
|
59
|
+
Define an asynchronous limiter that composes the pyrate limiter.
|
|
60
|
+
|
|
61
|
+
This class encapsulates the creation and configuration of a pyrate-limiter
|
|
62
|
+
Limiter with appropriate async bucket wrapper and settings.
|
|
63
|
+
|
|
64
|
+
Args:
|
|
65
|
+
limiter: An instance of an asynchronous pyrate limiter.
|
|
66
|
+
**kwargs: Additional keyword arguments for the parent classes.
|
|
67
|
+
|
|
68
|
+
"""
|
|
69
|
+
|
|
70
|
+
def __init__(self, *, limiter: Limiter, **kwargs: dict[str, object]) -> None:
|
|
71
|
+
super().__init__(**kwargs)
|
|
72
|
+
self._limiter = limiter
|
|
73
|
+
|
|
74
|
+
@classmethod
|
|
75
|
+
def create(
|
|
76
|
+
cls,
|
|
77
|
+
*rates: Rate,
|
|
78
|
+
**kwargs: Unpack[PyRateLimiterKeywordArguments],
|
|
79
|
+
) -> AsyncLimiter:
|
|
80
|
+
"""Create an instance of AsyncLimiter."""
|
|
81
|
+
if not rates:
|
|
82
|
+
msg = "At least one rate must be provided."
|
|
83
|
+
raise ValueError(msg)
|
|
84
|
+
|
|
85
|
+
rate_limits = [
|
|
86
|
+
PyRate(limit=rate.magnitude, interval=rate.in_milliseconds())
|
|
87
|
+
for rate in rates
|
|
88
|
+
]
|
|
89
|
+
if not validate_rate_list(rates=rate_limits):
|
|
90
|
+
url = "https://pyratelimiter.readthedocs.io/en/latest/#defining-rate-limits-and-buckets"
|
|
91
|
+
msg = (
|
|
92
|
+
f"Invalid ordering of rates provided {rate_limits}. Please read "
|
|
93
|
+
f"{url} for more information."
|
|
94
|
+
)
|
|
95
|
+
raise ValueError(msg)
|
|
96
|
+
|
|
97
|
+
limiter = Limiter(
|
|
98
|
+
argument=BucketAsyncWrapper(InMemoryBucket(rate_limits)),
|
|
99
|
+
clock=kwargs.get("clock", TimeAsyncClock()),
|
|
100
|
+
raise_when_fail=kwargs.get("raise_when_fail", False),
|
|
101
|
+
max_delay=kwargs.get("max_delay", Duration.HOUR),
|
|
102
|
+
retry_until_max_delay=kwargs.get("retry_until_max_delay", False),
|
|
103
|
+
buffer_ms=kwargs.get("buffer_ms", 50),
|
|
104
|
+
)
|
|
105
|
+
return cls(limiter=limiter)
|
|
106
|
+
|
|
107
|
+
async def __aenter__(self) -> AsyncLimiter: # noqa: PYI034
|
|
108
|
+
"""Acquire a token upon entering the asynchronous context."""
|
|
109
|
+
# Keep trying to acquire, let timeouts be handled externally.
|
|
110
|
+
while not (await self._limiter.try_acquire_async("httpx-limiter")):
|
|
111
|
+
pass # pragma: no cover
|
|
112
|
+
|
|
113
|
+
return self
|
|
114
|
+
|
|
115
|
+
async def __aexit__(
|
|
116
|
+
self,
|
|
117
|
+
_exc_type: type[BaseException] | None,
|
|
118
|
+
_exc_val: BaseException | None,
|
|
119
|
+
_exc_tb: TracebackType | None,
|
|
120
|
+
) -> None:
|
|
121
|
+
"""Exit the asynchronous context."""
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# Copyright (c) 2025 Moritz E. Beber
|
|
2
|
+
#
|
|
3
|
+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
|
4
|
+
# use this file except in compliance with the License. You may obtain a copy of
|
|
5
|
+
# the License at
|
|
6
|
+
#
|
|
7
|
+
# https://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
#
|
|
9
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
11
|
+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
12
|
+
# License for the specific language governing permissions and limitations under
|
|
13
|
+
# the License.
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
"""Provide an asynchronous multiple rate-limited transport."""
|
|
17
|
+
|
|
18
|
+
from __future__ import annotations
|
|
19
|
+
|
|
20
|
+
import sys
|
|
21
|
+
from typing import TYPE_CHECKING
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
if sys.version_info < (3, 11):
|
|
25
|
+
from typing_extensions import Unpack
|
|
26
|
+
else:
|
|
27
|
+
from typing import Unpack
|
|
28
|
+
|
|
29
|
+
import httpx
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
if TYPE_CHECKING: # pragma: no cover
|
|
33
|
+
from .abstract_rate_limiter_repository import AbstractRateLimiterRepository
|
|
34
|
+
from .types import HTTPXAsyncHTTPTransportKeywordArguments
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
class AsyncMultiRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
38
|
+
"""
|
|
39
|
+
Define the asynchronous multiple rate-limited transport.
|
|
40
|
+
|
|
41
|
+
This transport consists of a composed transport for handling requests and a
|
|
42
|
+
repository for rate limiters that are selected based on the request.
|
|
43
|
+
|
|
44
|
+
"""
|
|
45
|
+
|
|
46
|
+
def __init__(
|
|
47
|
+
self,
|
|
48
|
+
*,
|
|
49
|
+
repository: AbstractRateLimiterRepository,
|
|
50
|
+
transport: httpx.AsyncBaseTransport,
|
|
51
|
+
**kwargs: dict[str, object],
|
|
52
|
+
) -> None:
|
|
53
|
+
super().__init__(**kwargs)
|
|
54
|
+
self._repo = repository
|
|
55
|
+
self._transport = transport
|
|
56
|
+
|
|
57
|
+
@classmethod
|
|
58
|
+
def create(
|
|
59
|
+
cls,
|
|
60
|
+
*,
|
|
61
|
+
repository: AbstractRateLimiterRepository,
|
|
62
|
+
**kwargs: Unpack[HTTPXAsyncHTTPTransportKeywordArguments],
|
|
63
|
+
) -> AsyncMultiRateLimitedTransport:
|
|
64
|
+
"""
|
|
65
|
+
Create an instance of an asynchronous multiple rate-limited transport.
|
|
66
|
+
|
|
67
|
+
This factory method constructs the instance with an underlying
|
|
68
|
+
`httpx.AsyncHTTPTransport`.
|
|
69
|
+
That transport is passed any additional keyword arguments.
|
|
70
|
+
|
|
71
|
+
Args:
|
|
72
|
+
repository: The repository to use for rate limiters.
|
|
73
|
+
**kwargs: Additional keyword arguments are used in the construction of an
|
|
74
|
+
`httpx.AsyncHTTPTransport`.
|
|
75
|
+
|
|
76
|
+
Returns:
|
|
77
|
+
A default instance of the class created from the given arguments.
|
|
78
|
+
|
|
79
|
+
"""
|
|
80
|
+
return cls(
|
|
81
|
+
repository=repository,
|
|
82
|
+
transport=httpx.AsyncHTTPTransport(**kwargs), # type: ignore[arg-type]
|
|
83
|
+
)
|
|
84
|
+
|
|
85
|
+
async def handle_async_request(
|
|
86
|
+
self,
|
|
87
|
+
request: httpx.Request,
|
|
88
|
+
) -> httpx.Response:
|
|
89
|
+
"""Handle an asynchronous request with rate limiting."""
|
|
90
|
+
limiter = self._repo.get(request)
|
|
91
|
+
async with limiter:
|
|
92
|
+
return await self._transport.handle_async_request(request)
|
{httpx_limiter-0.2.0 → httpx_limiter-0.4.0}/src/httpx_limiter/async_rate_limited_transport.py
RENAMED
|
@@ -17,14 +17,23 @@
|
|
|
17
17
|
|
|
18
18
|
from __future__ import annotations
|
|
19
19
|
|
|
20
|
+
import sys
|
|
20
21
|
from typing import TYPE_CHECKING
|
|
21
22
|
|
|
23
|
+
|
|
24
|
+
if sys.version_info < (3, 11):
|
|
25
|
+
from typing_extensions import Unpack
|
|
26
|
+
else:
|
|
27
|
+
from typing import Unpack
|
|
28
|
+
|
|
22
29
|
import httpx
|
|
23
|
-
from aiolimiter import AsyncLimiter
|
|
24
30
|
|
|
31
|
+
from .async_limiter import AsyncLimiter
|
|
25
32
|
|
|
26
|
-
|
|
33
|
+
|
|
34
|
+
if TYPE_CHECKING: # pragma: no cover
|
|
27
35
|
from .rate import Rate
|
|
36
|
+
from .types import HTTPXAsyncHTTPTransportKeywordArguments
|
|
28
37
|
|
|
29
38
|
|
|
30
39
|
class AsyncRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
@@ -42,7 +51,7 @@ class AsyncRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
|
42
51
|
*,
|
|
43
52
|
limiter: AsyncLimiter,
|
|
44
53
|
transport: httpx.AsyncBaseTransport,
|
|
45
|
-
**kwargs,
|
|
54
|
+
**kwargs: dict[str, object],
|
|
46
55
|
) -> None:
|
|
47
56
|
super().__init__(**kwargs)
|
|
48
57
|
self._limiter = limiter
|
|
@@ -51,9 +60,8 @@ class AsyncRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
|
51
60
|
@classmethod
|
|
52
61
|
def create(
|
|
53
62
|
cls,
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
**kwargs: dict,
|
|
63
|
+
*rates: Rate,
|
|
64
|
+
**kwargs: Unpack[HTTPXAsyncHTTPTransportKeywordArguments],
|
|
57
65
|
) -> AsyncRateLimitedTransport:
|
|
58
66
|
"""
|
|
59
67
|
Create an instance of asynchronous rate-limited transport.
|
|
@@ -63,7 +71,7 @@ class AsyncRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
|
63
71
|
That transport is passed any additional keyword arguments.
|
|
64
72
|
|
|
65
73
|
Args:
|
|
66
|
-
|
|
74
|
+
*rates: One or more rate limits to apply.
|
|
67
75
|
**kwargs: Additional keyword arguments are used in the construction of an
|
|
68
76
|
`httpx.AsyncHTTPTransport`.
|
|
69
77
|
|
|
@@ -72,10 +80,7 @@ class AsyncRateLimitedTransport(httpx.AsyncBaseTransport):
|
|
|
72
80
|
|
|
73
81
|
"""
|
|
74
82
|
return cls(
|
|
75
|
-
limiter=AsyncLimiter(
|
|
76
|
-
max_rate=rate.magnitude,
|
|
77
|
-
time_period=rate.in_seconds(),
|
|
78
|
-
),
|
|
83
|
+
limiter=AsyncLimiter.create(*rates),
|
|
79
84
|
transport=httpx.AsyncHTTPTransport(**kwargs), # type: ignore[arg-type]
|
|
80
85
|
)
|
|
81
86
|
|
|
@@ -27,19 +27,27 @@ Number = float | int
|
|
|
27
27
|
class Rate(NamedTuple):
|
|
28
28
|
"""Define the rate."""
|
|
29
29
|
|
|
30
|
-
magnitude:
|
|
30
|
+
magnitude: int
|
|
31
31
|
duration: timedelta
|
|
32
32
|
|
|
33
33
|
@classmethod
|
|
34
|
-
def create(cls, magnitude:
|
|
34
|
+
def create(cls, magnitude: int = 1, duration: timedelta | Number = 1) -> Rate:
|
|
35
35
|
"""Create a rate."""
|
|
36
|
-
magnitude =
|
|
36
|
+
magnitude = int(magnitude)
|
|
37
|
+
|
|
38
|
+
if not (magnitude > 0):
|
|
39
|
+
msg = f"Magnitude must be strictly positive (provided {magnitude=})."
|
|
40
|
+
raise ValueError(msg)
|
|
37
41
|
|
|
38
42
|
if not isinstance(duration, timedelta):
|
|
39
43
|
duration = timedelta(seconds=float(duration))
|
|
40
44
|
|
|
45
|
+
if not (duration > timedelta(seconds=0)):
|
|
46
|
+
msg = f"Duration must be strictly positive (provided {duration=})."
|
|
47
|
+
raise ValueError(msg)
|
|
48
|
+
|
|
41
49
|
return cls(magnitude=magnitude, duration=duration)
|
|
42
50
|
|
|
43
|
-
def
|
|
44
|
-
"""Return the duration in unit
|
|
45
|
-
return self.duration.total_seconds()
|
|
51
|
+
def in_milliseconds(self) -> int:
|
|
52
|
+
"""Return the duration in unit milliseconds."""
|
|
53
|
+
return int(self.duration.total_seconds() * 1_000)
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Copyright (c) 2025 Moritz E. Beber
|
|
2
|
+
#
|
|
3
|
+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
|
4
|
+
# use this file except in compliance with the License. You may obtain a copy of
|
|
5
|
+
# the License at
|
|
6
|
+
#
|
|
7
|
+
# https://www.apache.org/licenses/LICENSE-2.0
|
|
8
|
+
#
|
|
9
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
10
|
+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
11
|
+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
12
|
+
# License for the specific language governing permissions and limitations under
|
|
13
|
+
# the License.
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
"""Provide additional type definitions."""
|
|
17
|
+
|
|
18
|
+
import ssl
|
|
19
|
+
import typing
|
|
20
|
+
from typing import TypedDict
|
|
21
|
+
|
|
22
|
+
import httpx
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
class HTTPXAsyncHTTPTransportKeywordArguments(TypedDict, total=False):
|
|
26
|
+
"""Keyword arguments for the httpx.AsyncHTTPTransport constructor."""
|
|
27
|
+
|
|
28
|
+
verify: ssl.SSLContext | str | bool
|
|
29
|
+
cert: httpx._types.CertTypes | None
|
|
30
|
+
trust_env: bool
|
|
31
|
+
http1: bool
|
|
32
|
+
http2: bool
|
|
33
|
+
limits: httpx._config.Limits
|
|
34
|
+
proxy: httpx._types.ProxyTypes | None
|
|
35
|
+
uds: str | None
|
|
36
|
+
local_address: str | None
|
|
37
|
+
retries: int
|
|
38
|
+
socket_options: typing.Iterable[httpx._transports.default.SOCKET_OPTION] | None
|
httpx_limiter-0.2.0/PKG-INFO
DELETED
|
@@ -1,73 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: httpx-limiter
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: A lightweight package that provides rate-limited httpx transports.
|
|
5
|
-
Project-URL: Homepage, https://github.com/Midnighter/httpx-limiter
|
|
6
|
-
Project-URL: Documentation, https://httpx-limiter.readthedocs.io
|
|
7
|
-
Project-URL: Source Code, https://github.com/Midnighter/httpx-limiter
|
|
8
|
-
Project-URL: Bug Tracker, https://github.com/Midnighter/httpx-limiter/issues
|
|
9
|
-
Project-URL: Download, https://pypi.org/project/httpx-limiter/#files
|
|
10
|
-
Author-email: "Moritz E. Beber" <midnighter@posteo.net>
|
|
11
|
-
License-Expression: Apache-2.0
|
|
12
|
-
License-File: LICENSE
|
|
13
|
-
Keywords: httpx,leaky bucket,limiter,rate-limit
|
|
14
|
-
Classifier: Development Status :: 2 - Pre-Alpha
|
|
15
|
-
Classifier: Environment :: Web Environment
|
|
16
|
-
Classifier: Framework :: AsyncIO
|
|
17
|
-
Classifier: Intended Audience :: Developers
|
|
18
|
-
Classifier: License :: OSI Approved :: Apache Software License
|
|
19
|
-
Classifier: Natural Language :: English
|
|
20
|
-
Classifier: Operating System :: OS Independent
|
|
21
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
22
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
23
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
24
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
25
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
26
|
-
Classifier: Topic :: Internet :: WWW/HTTP
|
|
27
|
-
Classifier: Typing :: Typed
|
|
28
|
-
Requires-Python: >=3.10
|
|
29
|
-
Requires-Dist: aiolimiter~=1.2
|
|
30
|
-
Requires-Dist: httpx~=0.25
|
|
31
|
-
Description-Content-Type: text/markdown
|
|
32
|
-
|
|
33
|
-
# HTTPX Limiter
|
|
34
|
-
|
|
35
|
-
| | |
|
|
36
|
-
|---|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
37
|
-
| Package | [](https://pypi.org/project/httpx-limiter/) [](https://pypi.org/project/httpx-limiter/) [](https://httpx-limiter.readthedocs.io/en/latest/?badge=latest) |
|
|
38
|
-
| Meta | [](LICENSE) [](.github/CODE_OF_CONDUCT.md) [](https://mypy-lang.org/) [](https://github.com/ambv/black) [](https://github.com/astral-sh/ruff) |
|
|
39
|
-
| Automation | |
|
|
40
|
-
|
|
41
|
-
_A lightweight package that provides rate-limited httpx transports._
|
|
42
|
-
|
|
43
|
-
## Installation
|
|
44
|
-
|
|
45
|
-
The package is published on [PyPI](https://pypi.org/project/httpx-limiter/).
|
|
46
|
-
Install it, for example, with
|
|
47
|
-
|
|
48
|
-
```sh
|
|
49
|
-
pip install httpx-limiter
|
|
50
|
-
```
|
|
51
|
-
|
|
52
|
-
## Usage
|
|
53
|
-
|
|
54
|
-
For situations when you need to make a large number of asynchronous request with
|
|
55
|
-
a controlled number of requests per unit time, you can apply rate limiting to an
|
|
56
|
-
HTTPX client using the provided transport. If you want to be able to make ten
|
|
57
|
-
requests per second, for example, use the following:
|
|
58
|
-
|
|
59
|
-
```python
|
|
60
|
-
import httpx
|
|
61
|
-
from httpx_limiter import AsyncRateLimitedTransport, Rate
|
|
62
|
-
|
|
63
|
-
async def main():
|
|
64
|
-
async with httpx.AsyncClient(
|
|
65
|
-
transport=AsyncRateLimitedTransport.create(rate=Rate.create(magnitude=10)),
|
|
66
|
-
) as client:
|
|
67
|
-
response = await client.get("https://httpbin.org")
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
## Copyright
|
|
71
|
-
|
|
72
|
-
- Copyright © 2024,2025 Moritz E. Beber.
|
|
73
|
-
- Free software distributed under the [Apache Software License 2.0](./LICENSE).
|
httpx_limiter-0.2.0/README.md
DELETED
|
@@ -1,41 +0,0 @@
|
|
|
1
|
-
# HTTPX Limiter
|
|
2
|
-
|
|
3
|
-
| | |
|
|
4
|
-
|---|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
5
|
-
| Package | [](https://pypi.org/project/httpx-limiter/) [](https://pypi.org/project/httpx-limiter/) [](https://httpx-limiter.readthedocs.io/en/latest/?badge=latest) |
|
|
6
|
-
| Meta | [](LICENSE) [](.github/CODE_OF_CONDUCT.md) [](https://mypy-lang.org/) [](https://github.com/ambv/black) [](https://github.com/astral-sh/ruff) |
|
|
7
|
-
| Automation | |
|
|
8
|
-
|
|
9
|
-
_A lightweight package that provides rate-limited httpx transports._
|
|
10
|
-
|
|
11
|
-
## Installation
|
|
12
|
-
|
|
13
|
-
The package is published on [PyPI](https://pypi.org/project/httpx-limiter/).
|
|
14
|
-
Install it, for example, with
|
|
15
|
-
|
|
16
|
-
```sh
|
|
17
|
-
pip install httpx-limiter
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Usage
|
|
21
|
-
|
|
22
|
-
For situations when you need to make a large number of asynchronous request with
|
|
23
|
-
a controlled number of requests per unit time, you can apply rate limiting to an
|
|
24
|
-
HTTPX client using the provided transport. If you want to be able to make ten
|
|
25
|
-
requests per second, for example, use the following:
|
|
26
|
-
|
|
27
|
-
```python
|
|
28
|
-
import httpx
|
|
29
|
-
from httpx_limiter import AsyncRateLimitedTransport, Rate
|
|
30
|
-
|
|
31
|
-
async def main():
|
|
32
|
-
async with httpx.AsyncClient(
|
|
33
|
-
transport=AsyncRateLimitedTransport.create(rate=Rate.create(magnitude=10)),
|
|
34
|
-
) as client:
|
|
35
|
-
response = await client.get("https://httpbin.org")
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
## Copyright
|
|
39
|
-
|
|
40
|
-
- Copyright © 2024,2025 Moritz E. Beber.
|
|
41
|
-
- Free software distributed under the [Apache Software License 2.0](./LICENSE).
|
|
File without changes
|
|
File without changes
|
|
File without changes
|