wexample-queue 1.0.1__tar.gz → 1.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {wexample_queue-1.0.1 → wexample_queue-1.1.0}/PKG-INFO +108 -5
- wexample_queue-1.1.0/README.md +243 -0
- {wexample_queue-1.0.1 → wexample_queue-1.1.0}/pyproject.toml +6 -2
- wexample_queue-1.1.0/src/wexample_queue/connector/rabbitmq_external_connector.py +127 -0
- wexample_queue-1.1.0/src/wexample_queue/service/__init__.py +0 -0
- wexample_queue-1.1.0/src/wexample_queue/service/abstract_queue_service.py +99 -0
- wexample_queue-1.1.0/tests/.gitkeep +0 -0
- wexample_queue-1.0.1/README.md +0 -143
- {wexample_queue-1.0.1 → wexample_queue-1.1.0}/src/wexample_queue/__init__.py +0 -0
- /wexample_queue-1.0.1/src/wexample_queue/py.typed → /wexample_queue-1.1.0/src/wexample_queue/connector/__init__.py +0 -0
- /wexample_queue-1.0.1/tests/.gitkeep → /wexample_queue-1.1.0/src/wexample_queue/py.typed +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: wexample-queue
|
|
3
|
-
Version: 1.0
|
|
3
|
+
Version: 1.1.0
|
|
4
4
|
Summary: Manage Queuing
|
|
5
5
|
Author-Email: weeger <contact@wexample.com>
|
|
6
6
|
License: MIT
|
|
@@ -8,6 +8,9 @@ Classifier: Programming Language :: Python :: 3
|
|
|
8
8
|
Classifier: License :: OSI Approved :: MIT License
|
|
9
9
|
Classifier: Operating System :: OS Independent
|
|
10
10
|
Requires-Python: >=3.10
|
|
11
|
+
Requires-Dist: pika>=1.3.0
|
|
12
|
+
Requires-Dist: wexample-app>=19.2.0
|
|
13
|
+
Requires-Dist: wexample-helpers>=20.0.0
|
|
11
14
|
Provides-Extra: dev
|
|
12
15
|
Requires-Dist: pytest; extra == "dev"
|
|
13
16
|
Requires-Dist: pytest-cov; extra == "dev"
|
|
@@ -15,15 +18,66 @@ Description-Content-Type: text/markdown
|
|
|
15
18
|
|
|
16
19
|
# queue
|
|
17
20
|
|
|
18
|
-
Version: 1.0
|
|
21
|
+
Version: 1.1.0
|
|
19
22
|
|
|
20
|
-
|
|
23
|
+
`wexample-queue` is what a worker needs to live on a RabbitMQ queue and nothing more: a connector holding the broker connection, and a service base whose whole subject is one queue. A subclass writes the name of the queue and what to do with a message; connecting, declaring, decoding, acknowledging and giving up are the same for every worker and are already written.
|
|
24
|
+
|
|
25
|
+
It exists because the other end of the queue is not always Python. The body of a message is plain JSON, published on the default exchange with the queue name as routing key — which is where a Symfony Messenger transport declaring that same queue binds its own exchange, so both ends meet in the same place without either knowing the other.
|
|
26
|
+
|
|
27
|
+
## A worker
|
|
28
|
+
|
|
29
|
+
```python
|
|
30
|
+
from wexample_queue.service.abstract_queue_service import AbstractQueueService
|
|
31
|
+
|
|
32
|
+
@base_class
|
|
33
|
+
class ProcessRunService(AbstractQueueService):
|
|
34
|
+
queue_name: str = public_field(
|
|
35
|
+
default="process_run",
|
|
36
|
+
description="Name of the queue this service consumes",
|
|
37
|
+
)
|
|
38
|
+
|
|
39
|
+
def process_message(self, message: dict[str, Any]) -> None:
|
|
40
|
+
self.log(f"Run {message['id']} asked for in {message['workdir']}")
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Calling `start()` opens the connection, declares the queue and blocks consuming
|
|
44
|
+
until `stop()`. Returning from `process_message()` acknowledges the message;
|
|
45
|
+
raising rejects it without putting it back, because requeueing a message that
|
|
46
|
+
just failed is asking for the same failure at once and for ever. Where it goes
|
|
47
|
+
then is the broker's business — a dead letter exchange if the queue has one,
|
|
48
|
+
nowhere if it has not.
|
|
49
|
+
|
|
50
|
+
## Publishing
|
|
51
|
+
|
|
52
|
+
```python
|
|
53
|
+
self.connector.publish("process_run_event", {"kind": "process_run", "id": run_id})
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## The broker
|
|
57
|
+
|
|
58
|
+
Where the broker is comes from the environment, under four keys the connector
|
|
59
|
+
declares as expected:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
RABBITMQ_HOST
|
|
63
|
+
RABBITMQ_PORT
|
|
64
|
+
RABBITMQ_USER
|
|
65
|
+
RABBITMQ_PASSWORD
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Reading them from there rather than from a caller is what keeps a worker and
|
|
69
|
+
whoever publishes to it from holding two addresses that could disagree.
|
|
21
70
|
|
|
22
71
|
## Table of Contents
|
|
23
72
|
|
|
73
|
+
- [A worker](#a-worker)
|
|
74
|
+
- [Publishing](#publishing)
|
|
75
|
+
- [The broker](#the-broker)
|
|
24
76
|
- [Installation](#installation)
|
|
25
77
|
- [Tests](#tests)
|
|
78
|
+
- [Architecture](#architecture)
|
|
26
79
|
- [Integration in the Suite](#integration-in-the-suite)
|
|
80
|
+
- [Dependencies](#dependencies)
|
|
27
81
|
- [Versioning & Compatibility Policy](#versioning--compatibility-policy)
|
|
28
82
|
- [License](#license)
|
|
29
83
|
- [About us](#about-us)
|
|
@@ -40,7 +94,34 @@ pip install wexample-queue
|
|
|
40
94
|
|
|
41
95
|
Requires Python >=3.10.
|
|
42
96
|
|
|
43
|
-
|
|
97
|
+
```bash
|
|
98
|
+
pip install wexample-queue
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
A worker is a class with a queue name and a `process_message()`:
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
from typing import Any
|
|
105
|
+
|
|
106
|
+
from wexample_helpers.classes.field import public_field
|
|
107
|
+
from wexample_helpers.decorator.base_class import base_class
|
|
108
|
+
from wexample_queue.service.abstract_queue_service import AbstractQueueService
|
|
109
|
+
|
|
110
|
+
@base_class
|
|
111
|
+
class HelloService(AbstractQueueService):
|
|
112
|
+
queue_name: str = public_field(
|
|
113
|
+
default="hello",
|
|
114
|
+
description="Name of the queue this service consumes",
|
|
115
|
+
)
|
|
116
|
+
|
|
117
|
+
def process_message(self, message: dict[str, Any]) -> None:
|
|
118
|
+
self.log(f"Got {message}")
|
|
119
|
+
|
|
120
|
+
HelloService(kernel=kernel, queue_name="hello").start()
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
`start()` blocks. The four `RABBITMQ_*` environment keys have to be readable by
|
|
124
|
+
the kernel, which is where the connector goes to find the broker.
|
|
44
125
|
|
|
45
126
|
## Tests
|
|
46
127
|
|
|
@@ -91,7 +172,23 @@ To enforce a minimum coverage percentage:
|
|
|
91
172
|
|
|
92
173
|
This will cause the test suite to fail if coverage drops below 80%.
|
|
93
174
|
|
|
94
|
-
|
|
175
|
+
## Architecture
|
|
176
|
+
|
|
177
|
+
Two classes, and the line between them is what a broker is against what a worker is.
|
|
178
|
+
|
|
179
|
+
### The connector
|
|
180
|
+
|
|
181
|
+
src/wexample_queue/connector/rabbitmq_external_connector.py extends `AbstractExternalConnector` from `wexample-app` and holds a `pika.BlockingConnection` and its channel. It exposes only what this package needs — `connect`, `disconnect`, `declare_queue`, `publish`, `consume`, `stop_consuming` — and reads the broker address from four environment keys it declares in `get_expected_env_keys()`, so a worker never carries one.
|
|
182
|
+
|
|
183
|
+
`publish()` goes through the default exchange with the queue name as routing key, and marks the message persistent. `declare_queue()` declares it durable, with the same arguments both ends use: neither side depends on the other having started first, and a broker restarted on its own finds its queues again.
|
|
184
|
+
|
|
185
|
+
### The service
|
|
186
|
+
|
|
187
|
+
src/wexample_queue/service/abstract_queue_service.py extends `AbstractService` from `wexample-app`, which already brings the run loop, the signal handling and the shutdown. What this adds is the queue: `start()` opens the connection and declares the queue before handing over to the loop, `_run()` blocks consuming, and `_on_message()` decodes one delivery and answers the broker for it.
|
|
188
|
+
|
|
189
|
+
The answer is the part worth reading. A body that is not JSON, and a message `process_message()` raised on, are both rejected with `requeue=False`. Putting such a message back would be asking for the same failure immediately and for ever; dropping it silently with an `ack` would lose it without saying so. Rejecting it hands the decision to the broker, which is the only place a dead letter exchange can be configured.
|
|
190
|
+
|
|
191
|
+
Unlike the implementation this was drawn from, constructing a service does not start it. `start()` is called by whoever owns the worker, which is what makes a service testable without a broker.
|
|
95
192
|
|
|
96
193
|
## Integration in the Suite
|
|
97
194
|
|
|
@@ -103,6 +200,12 @@ The suite includes packages for configuration management, file handling, prompts
|
|
|
103
200
|
|
|
104
201
|
Visit the [Wexample Suite documentation](https://docs.wexample.com) for the complete package ecosystem.
|
|
105
202
|
|
|
203
|
+
## Dependencies
|
|
204
|
+
|
|
205
|
+
- pika: >=1.3.0
|
|
206
|
+
- wexample-app: >=19.2.0
|
|
207
|
+
- wexample-helpers: >=20.0.0
|
|
208
|
+
|
|
106
209
|
## Versioning & Compatibility Policy
|
|
107
210
|
|
|
108
211
|
Wexample packages follow **Semantic Versioning** (SemVer):
|
|
@@ -0,0 +1,243 @@
|
|
|
1
|
+
# queue
|
|
2
|
+
|
|
3
|
+
Version: 1.1.0
|
|
4
|
+
|
|
5
|
+
`wexample-queue` is what a worker needs to live on a RabbitMQ queue and nothing more: a connector holding the broker connection, and a service base whose whole subject is one queue. A subclass writes the name of the queue and what to do with a message; connecting, declaring, decoding, acknowledging and giving up are the same for every worker and are already written.
|
|
6
|
+
|
|
7
|
+
It exists because the other end of the queue is not always Python. The body of a message is plain JSON, published on the default exchange with the queue name as routing key — which is where a Symfony Messenger transport declaring that same queue binds its own exchange, so both ends meet in the same place without either knowing the other.
|
|
8
|
+
|
|
9
|
+
## A worker
|
|
10
|
+
|
|
11
|
+
```python
|
|
12
|
+
from wexample_queue.service.abstract_queue_service import AbstractQueueService
|
|
13
|
+
|
|
14
|
+
@base_class
|
|
15
|
+
class ProcessRunService(AbstractQueueService):
|
|
16
|
+
queue_name: str = public_field(
|
|
17
|
+
default="process_run",
|
|
18
|
+
description="Name of the queue this service consumes",
|
|
19
|
+
)
|
|
20
|
+
|
|
21
|
+
def process_message(self, message: dict[str, Any]) -> None:
|
|
22
|
+
self.log(f"Run {message['id']} asked for in {message['workdir']}")
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Calling `start()` opens the connection, declares the queue and blocks consuming
|
|
26
|
+
until `stop()`. Returning from `process_message()` acknowledges the message;
|
|
27
|
+
raising rejects it without putting it back, because requeueing a message that
|
|
28
|
+
just failed is asking for the same failure at once and for ever. Where it goes
|
|
29
|
+
then is the broker's business — a dead letter exchange if the queue has one,
|
|
30
|
+
nowhere if it has not.
|
|
31
|
+
|
|
32
|
+
## Publishing
|
|
33
|
+
|
|
34
|
+
```python
|
|
35
|
+
self.connector.publish("process_run_event", {"kind": "process_run", "id": run_id})
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## The broker
|
|
39
|
+
|
|
40
|
+
Where the broker is comes from the environment, under four keys the connector
|
|
41
|
+
declares as expected:
|
|
42
|
+
|
|
43
|
+
```
|
|
44
|
+
RABBITMQ_HOST
|
|
45
|
+
RABBITMQ_PORT
|
|
46
|
+
RABBITMQ_USER
|
|
47
|
+
RABBITMQ_PASSWORD
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Reading them from there rather than from a caller is what keeps a worker and
|
|
51
|
+
whoever publishes to it from holding two addresses that could disagree.
|
|
52
|
+
|
|
53
|
+
## Table of Contents
|
|
54
|
+
|
|
55
|
+
- [A worker](#a-worker)
|
|
56
|
+
- [Publishing](#publishing)
|
|
57
|
+
- [The broker](#the-broker)
|
|
58
|
+
- [Installation](#installation)
|
|
59
|
+
- [Tests](#tests)
|
|
60
|
+
- [Architecture](#architecture)
|
|
61
|
+
- [Integration in the Suite](#integration-in-the-suite)
|
|
62
|
+
- [Dependencies](#dependencies)
|
|
63
|
+
- [Versioning & Compatibility Policy](#versioning--compatibility-policy)
|
|
64
|
+
- [License](#license)
|
|
65
|
+
- [About us](#about-us)
|
|
66
|
+
- [Known Limitations & Roadmap](#known-limitations--roadmap)
|
|
67
|
+
- [Status & Compatibility](#status--compatibility)
|
|
68
|
+
- [Useful Links](#useful-links)
|
|
69
|
+
- [Migration Notes](#migration-notes)
|
|
70
|
+
|
|
71
|
+
## Installation
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
pip install wexample-queue
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Requires Python >=3.10.
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
pip install wexample-queue
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
A worker is a class with a queue name and a `process_message()`:
|
|
84
|
+
|
|
85
|
+
```python
|
|
86
|
+
from typing import Any
|
|
87
|
+
|
|
88
|
+
from wexample_helpers.classes.field import public_field
|
|
89
|
+
from wexample_helpers.decorator.base_class import base_class
|
|
90
|
+
from wexample_queue.service.abstract_queue_service import AbstractQueueService
|
|
91
|
+
|
|
92
|
+
@base_class
|
|
93
|
+
class HelloService(AbstractQueueService):
|
|
94
|
+
queue_name: str = public_field(
|
|
95
|
+
default="hello",
|
|
96
|
+
description="Name of the queue this service consumes",
|
|
97
|
+
)
|
|
98
|
+
|
|
99
|
+
def process_message(self, message: dict[str, Any]) -> None:
|
|
100
|
+
self.log(f"Got {message}")
|
|
101
|
+
|
|
102
|
+
HelloService(kernel=kernel, queue_name="hello").start()
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
`start()` blocks. The four `RABBITMQ_*` environment keys have to be readable by
|
|
106
|
+
the kernel, which is where the connector goes to find the broker.
|
|
107
|
+
|
|
108
|
+
## Tests
|
|
109
|
+
|
|
110
|
+
This project uses `pytest` for testing and `pytest-cov` for code coverage analysis.
|
|
111
|
+
|
|
112
|
+
### Installation
|
|
113
|
+
|
|
114
|
+
First, install the required testing dependencies:
|
|
115
|
+
```bash
|
|
116
|
+
.venv/bin/python -m pip install pytest pytest-cov
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
### Basic Usage
|
|
120
|
+
|
|
121
|
+
Run all tests with coverage:
|
|
122
|
+
```bash
|
|
123
|
+
.venv/bin/python -m pytest --cov --cov-report=html
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
### Common Commands
|
|
127
|
+
```bash
|
|
128
|
+
# Run tests with coverage for a specific module
|
|
129
|
+
.venv/bin/python -m pytest --cov=your_module
|
|
130
|
+
|
|
131
|
+
# Show which lines are not covered
|
|
132
|
+
.venv/bin/python -m pytest --cov=your_module --cov-report=term-missing
|
|
133
|
+
|
|
134
|
+
# Generate an HTML coverage report
|
|
135
|
+
.venv/bin/python -m pytest --cov=your_module --cov-report=html
|
|
136
|
+
|
|
137
|
+
# Combine terminal and HTML reports
|
|
138
|
+
.venv/bin/python -m pytest --cov=your_module --cov-report=term-missing --cov-report=html
|
|
139
|
+
|
|
140
|
+
# Run specific test file with coverage
|
|
141
|
+
.venv/bin/python -m pytest tests/test_file.py --cov=your_module --cov-report=term-missing
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
### Viewing HTML Reports
|
|
145
|
+
|
|
146
|
+
After generating an HTML report, open `htmlcov/index.html` in your browser to view detailed line-by-line coverage information.
|
|
147
|
+
|
|
148
|
+
### Coverage Threshold
|
|
149
|
+
|
|
150
|
+
To enforce a minimum coverage percentage:
|
|
151
|
+
```bash
|
|
152
|
+
.venv/bin/python -m pytest --cov=your_module --cov-fail-under=80
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
This will cause the test suite to fail if coverage drops below 80%.
|
|
156
|
+
|
|
157
|
+
## Architecture
|
|
158
|
+
|
|
159
|
+
Two classes, and the line between them is what a broker is against what a worker is.
|
|
160
|
+
|
|
161
|
+
### The connector
|
|
162
|
+
|
|
163
|
+
src/wexample_queue/connector/rabbitmq_external_connector.py extends `AbstractExternalConnector` from `wexample-app` and holds a `pika.BlockingConnection` and its channel. It exposes only what this package needs — `connect`, `disconnect`, `declare_queue`, `publish`, `consume`, `stop_consuming` — and reads the broker address from four environment keys it declares in `get_expected_env_keys()`, so a worker never carries one.
|
|
164
|
+
|
|
165
|
+
`publish()` goes through the default exchange with the queue name as routing key, and marks the message persistent. `declare_queue()` declares it durable, with the same arguments both ends use: neither side depends on the other having started first, and a broker restarted on its own finds its queues again.
|
|
166
|
+
|
|
167
|
+
### The service
|
|
168
|
+
|
|
169
|
+
src/wexample_queue/service/abstract_queue_service.py extends `AbstractService` from `wexample-app`, which already brings the run loop, the signal handling and the shutdown. What this adds is the queue: `start()` opens the connection and declares the queue before handing over to the loop, `_run()` blocks consuming, and `_on_message()` decodes one delivery and answers the broker for it.
|
|
170
|
+
|
|
171
|
+
The answer is the part worth reading. A body that is not JSON, and a message `process_message()` raised on, are both rejected with `requeue=False`. Putting such a message back would be asking for the same failure immediately and for ever; dropping it silently with an `ack` would lose it without saying so. Rejecting it hands the decision to the broker, which is the only place a dead letter exchange can be configured.
|
|
172
|
+
|
|
173
|
+
Unlike the implementation this was drawn from, constructing a service does not start it. `start()` is called by whoever owns the worker, which is what makes a service testable without a broker.
|
|
174
|
+
|
|
175
|
+
## Integration in the Suite
|
|
176
|
+
|
|
177
|
+
This package is part of the Wexample Suite — a collection of high-quality, modular tools designed to work seamlessly together across multiple languages and environments.
|
|
178
|
+
|
|
179
|
+
### Related Packages
|
|
180
|
+
|
|
181
|
+
The suite includes packages for configuration management, file handling, prompts, and more. Each package can be used independently or as part of the integrated suite.
|
|
182
|
+
|
|
183
|
+
Visit the [Wexample Suite documentation](https://docs.wexample.com) for the complete package ecosystem.
|
|
184
|
+
|
|
185
|
+
## Dependencies
|
|
186
|
+
|
|
187
|
+
- pika: >=1.3.0
|
|
188
|
+
- wexample-app: >=19.2.0
|
|
189
|
+
- wexample-helpers: >=20.0.0
|
|
190
|
+
|
|
191
|
+
## Versioning & Compatibility Policy
|
|
192
|
+
|
|
193
|
+
Wexample packages follow **Semantic Versioning** (SemVer):
|
|
194
|
+
|
|
195
|
+
- **MAJOR**: Breaking changes
|
|
196
|
+
- **MINOR**: New features, backward compatible
|
|
197
|
+
- **PATCH**: Bug fixes, backward compatible
|
|
198
|
+
|
|
199
|
+
We maintain backward compatibility within major versions and provide clear migration guides for breaking changes.
|
|
200
|
+
|
|
201
|
+
## License
|
|
202
|
+
|
|
203
|
+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
204
|
+
|
|
205
|
+
Free to use in both personal and commercial projects.
|
|
206
|
+
|
|
207
|
+
## About us
|
|
208
|
+
|
|
209
|
+
[Wexample](https://wexample.com) stands as a cornerstone of the digital ecosystem — a collective of seasoned engineers, researchers, and creators driven by a relentless pursuit of technological excellence. More than a media platform, it has grown into a vibrant community where innovation meets craftsmanship, and where every line of code reflects a commitment to clarity, durability, and shared intelligence.
|
|
210
|
+
|
|
211
|
+
This packages suite embodies this spirit. Trusted by professionals and enthusiasts alike, it delivers a consistent, high-quality foundation for modern development — open, elegant, and battle-tested. Its reputation is built on years of collaboration, refinement, and rigorous attention to detail, making it a natural choice for those who demand both robustness and beauty in their tools.
|
|
212
|
+
|
|
213
|
+
Wexample cultivates a culture of mastery. Each package, each contribution carries the mark of a community that values precision, ethics, and innovation — a community proud to shape the future of digital craftsmanship.
|
|
214
|
+
|
|
215
|
+
## Known Limitations & Roadmap
|
|
216
|
+
|
|
217
|
+
Current limitations and planned features are tracked in the GitHub issues.
|
|
218
|
+
|
|
219
|
+
See the [project roadmap](https://github.com/wexample/python-queue/issues) for upcoming features and improvements.
|
|
220
|
+
|
|
221
|
+
## Status & Compatibility
|
|
222
|
+
|
|
223
|
+
**Maturity**: Production-ready
|
|
224
|
+
|
|
225
|
+
**Python Support**: >=3.10
|
|
226
|
+
|
|
227
|
+
**OS Support**: Linux, macOS, Windows
|
|
228
|
+
|
|
229
|
+
**Status**: Actively maintained
|
|
230
|
+
|
|
231
|
+
## Useful Links
|
|
232
|
+
|
|
233
|
+
- **Homepage**: https://github.com/wexample/python-queue
|
|
234
|
+
- **Documentation**: [docs.wexample.com](https://docs.wexample.com)
|
|
235
|
+
- **Issue Tracker**: https://github.com/wexample/python-queue/issues
|
|
236
|
+
- **Discussions**: https://github.com/wexample/python-queue/discussions
|
|
237
|
+
- **PyPI**: [pypi.org/project/wexample-queue](https://pypi.org/project/wexample-queue/)
|
|
238
|
+
|
|
239
|
+
## Migration Notes
|
|
240
|
+
|
|
241
|
+
When upgrading between major versions, refer to the migration guides in the documentation.
|
|
242
|
+
|
|
243
|
+
Breaking changes are clearly documented with upgrade paths and examples.
|
|
@@ -6,7 +6,7 @@ build-backend = "pdm.backend"
|
|
|
6
6
|
|
|
7
7
|
[project]
|
|
8
8
|
name = "wexample-queue"
|
|
9
|
-
version = "1.0
|
|
9
|
+
version = "1.1.0"
|
|
10
10
|
description = "Manage Queuing"
|
|
11
11
|
authors = [
|
|
12
12
|
{ name = "weeger", email = "contact@wexample.com" },
|
|
@@ -17,7 +17,11 @@ classifiers = [
|
|
|
17
17
|
"License :: OSI Approved :: MIT License",
|
|
18
18
|
"Operating System :: OS Independent",
|
|
19
19
|
]
|
|
20
|
-
dependencies = [
|
|
20
|
+
dependencies = [
|
|
21
|
+
"pika>=1.3.0",
|
|
22
|
+
"wexample-app>=19.2.0",
|
|
23
|
+
"wexample-helpers>=20.0.0",
|
|
24
|
+
]
|
|
21
25
|
|
|
22
26
|
[project.readme]
|
|
23
27
|
file = "README.md"
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
from typing import TYPE_CHECKING, Any
|
|
4
|
+
|
|
5
|
+
import pika
|
|
6
|
+
from wexample_app.connector.abstract_external_connector import (
|
|
7
|
+
AbstractExternalConnector,
|
|
8
|
+
)
|
|
9
|
+
from wexample_helpers.classes.field import public_field
|
|
10
|
+
from wexample_helpers.decorator.base_class import base_class
|
|
11
|
+
|
|
12
|
+
if TYPE_CHECKING:
|
|
13
|
+
from collections.abc import Callable
|
|
14
|
+
|
|
15
|
+
from wexample_helpers.const.types import StringsList
|
|
16
|
+
|
|
17
|
+
RABBITMQ_ENV_KEY_HOST = "RABBITMQ_HOST"
|
|
18
|
+
RABBITMQ_ENV_KEY_PASSWORD = "RABBITMQ_PASSWORD"
|
|
19
|
+
RABBITMQ_ENV_KEY_PORT = "RABBITMQ_PORT"
|
|
20
|
+
RABBITMQ_ENV_KEY_USER = "RABBITMQ_USER"
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
@base_class
|
|
24
|
+
class RabbitmqExternalConnector(AbstractExternalConnector):
|
|
25
|
+
"""A connection to a RabbitMQ broker, and the few things one does with it.
|
|
26
|
+
|
|
27
|
+
Where the broker is comes from the environment and not from a caller: a
|
|
28
|
+
worker and whoever publishes to it read the same four keys, so neither of
|
|
29
|
+
them holds an address the other could contradict.
|
|
30
|
+
"""
|
|
31
|
+
|
|
32
|
+
channel: Any = public_field(
|
|
33
|
+
default=None,
|
|
34
|
+
description="The channel opened on the connection, once connected",
|
|
35
|
+
)
|
|
36
|
+
connected: bool = public_field(
|
|
37
|
+
default=False,
|
|
38
|
+
description="Whether the connection is currently open",
|
|
39
|
+
)
|
|
40
|
+
connection: Any = public_field(
|
|
41
|
+
default=None,
|
|
42
|
+
description="The blocking connection to the broker, once connected",
|
|
43
|
+
)
|
|
44
|
+
name: str = public_field(
|
|
45
|
+
default="rabbitmq",
|
|
46
|
+
description="Name this connector answers to",
|
|
47
|
+
)
|
|
48
|
+
|
|
49
|
+
def connect(self) -> None:
|
|
50
|
+
if self.connected:
|
|
51
|
+
return
|
|
52
|
+
|
|
53
|
+
self.connection = pika.BlockingConnection(
|
|
54
|
+
pika.ConnectionParameters(
|
|
55
|
+
host=self._env(RABBITMQ_ENV_KEY_HOST),
|
|
56
|
+
port=int(self._env(RABBITMQ_ENV_KEY_PORT)),
|
|
57
|
+
credentials=pika.PlainCredentials(
|
|
58
|
+
self._env(RABBITMQ_ENV_KEY_USER),
|
|
59
|
+
self._env(RABBITMQ_ENV_KEY_PASSWORD),
|
|
60
|
+
),
|
|
61
|
+
)
|
|
62
|
+
)
|
|
63
|
+
self.channel = self.connection.channel()
|
|
64
|
+
self.connected = True
|
|
65
|
+
|
|
66
|
+
def consume(self, queue_name: str, callback: Callable) -> None:
|
|
67
|
+
"""Hands the channel over to the callback until consumption is stopped."""
|
|
68
|
+
self.channel.basic_consume(
|
|
69
|
+
queue=queue_name, on_message_callback=callback
|
|
70
|
+
)
|
|
71
|
+
self.channel.start_consuming()
|
|
72
|
+
|
|
73
|
+
def declare_queue(self, queue_name: str) -> None:
|
|
74
|
+
"""Makes sure the queue is there, durable, whoever gets there first.
|
|
75
|
+
|
|
76
|
+
Both ends declare it with the same arguments so that neither depends on
|
|
77
|
+
the other having started, and a broker restarted on its own keeps the
|
|
78
|
+
queue rather than the messages in it.
|
|
79
|
+
"""
|
|
80
|
+
self.channel.queue_declare(queue=queue_name, durable=True)
|
|
81
|
+
|
|
82
|
+
def disconnect(self) -> None:
|
|
83
|
+
if not self.connected:
|
|
84
|
+
return
|
|
85
|
+
|
|
86
|
+
self.connection.close()
|
|
87
|
+
self.connection = None
|
|
88
|
+
self.channel = None
|
|
89
|
+
self.connected = False
|
|
90
|
+
|
|
91
|
+
def get_expected_env_keys(self) -> StringsList:
|
|
92
|
+
return [
|
|
93
|
+
RABBITMQ_ENV_KEY_HOST,
|
|
94
|
+
RABBITMQ_ENV_KEY_PORT,
|
|
95
|
+
RABBITMQ_ENV_KEY_USER,
|
|
96
|
+
RABBITMQ_ENV_KEY_PASSWORD,
|
|
97
|
+
]
|
|
98
|
+
|
|
99
|
+
def publish(self, queue_name: str, payload: dict[str, Any]) -> None:
|
|
100
|
+
"""Puts one message on the queue, marked to survive a broker restart.
|
|
101
|
+
|
|
102
|
+
It goes through the default exchange, whose routing key is the queue
|
|
103
|
+
name — which is also where a Symfony Messenger transport declaring that
|
|
104
|
+
queue binds its own exchange, so both land in the same place.
|
|
105
|
+
"""
|
|
106
|
+
import json
|
|
107
|
+
|
|
108
|
+
self.channel.basic_publish(
|
|
109
|
+
exchange="",
|
|
110
|
+
routing_key=queue_name,
|
|
111
|
+
body=json.dumps(payload),
|
|
112
|
+
properties=pika.BasicProperties(delivery_mode=2),
|
|
113
|
+
)
|
|
114
|
+
|
|
115
|
+
def stop_consuming(self) -> None:
|
|
116
|
+
self.channel.stop_consuming()
|
|
117
|
+
|
|
118
|
+
def _env(self, key: str) -> str:
|
|
119
|
+
"""One of the four keys, from the process environment or from wex's own.
|
|
120
|
+
|
|
121
|
+
Both are read because both are how a worker is configured: a container
|
|
122
|
+
gets its broker from its environment, and a worker started by hand gets
|
|
123
|
+
it from the app's env file.
|
|
124
|
+
"""
|
|
125
|
+
import os
|
|
126
|
+
|
|
127
|
+
return os.environ.get(key) or self.kernel.get_env_parameter(key)
|
|
File without changes
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
from typing import TYPE_CHECKING, Any
|
|
4
|
+
|
|
5
|
+
from wexample_app.service.abstract_service import AbstractService
|
|
6
|
+
from wexample_helpers.classes.abstract_method import abstract_method
|
|
7
|
+
from wexample_helpers.classes.field import public_field
|
|
8
|
+
from wexample_helpers.decorator.base_class import base_class
|
|
9
|
+
|
|
10
|
+
if TYPE_CHECKING:
|
|
11
|
+
from wexample_queue.connector.rabbitmq_external_connector import (
|
|
12
|
+
RabbitmqExternalConnector,
|
|
13
|
+
)
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
@base_class
|
|
17
|
+
class AbstractQueueService(AbstractService):
|
|
18
|
+
"""A service whose work is whatever arrives on one queue.
|
|
19
|
+
|
|
20
|
+
All a subclass writes is the name of the queue and what to do with a
|
|
21
|
+
message. Connecting, declaring, decoding, acknowledging and giving up are
|
|
22
|
+
the same for every worker, so they are here.
|
|
23
|
+
"""
|
|
24
|
+
|
|
25
|
+
connector: RabbitmqExternalConnector | None = public_field(
|
|
26
|
+
default=None,
|
|
27
|
+
description="The broker connection, opened when the service starts",
|
|
28
|
+
)
|
|
29
|
+
queue_name: str = public_field(
|
|
30
|
+
description="Name of the queue this service consumes",
|
|
31
|
+
)
|
|
32
|
+
|
|
33
|
+
@abstract_method
|
|
34
|
+
def process_message(self, message: dict[str, Any]) -> None:
|
|
35
|
+
"""What to do with one message, already decoded.
|
|
36
|
+
|
|
37
|
+
Returning means the message is acknowledged. Raising means it is given
|
|
38
|
+
up on — so a subclass that wants a message tried again has to say so by
|
|
39
|
+
putting it back itself.
|
|
40
|
+
"""
|
|
41
|
+
|
|
42
|
+
def start(self) -> None:
|
|
43
|
+
from wexample_queue.connector.rabbitmq_external_connector import (
|
|
44
|
+
RabbitmqExternalConnector,
|
|
45
|
+
)
|
|
46
|
+
|
|
47
|
+
self.connector = RabbitmqExternalConnector(kernel=self.kernel)
|
|
48
|
+
self.connector.connect()
|
|
49
|
+
self.connector.declare_queue(self.queue_name)
|
|
50
|
+
|
|
51
|
+
super().start()
|
|
52
|
+
|
|
53
|
+
def stop(self) -> None:
|
|
54
|
+
if self.connector:
|
|
55
|
+
self.connector.stop_consuming()
|
|
56
|
+
self.connector.disconnect()
|
|
57
|
+
|
|
58
|
+
super().stop()
|
|
59
|
+
|
|
60
|
+
def _on_message(self, channel, method_frame, header_frame, body) -> None:
|
|
61
|
+
"""Decodes one delivery and answers the broker for it.
|
|
62
|
+
|
|
63
|
+
A message that cannot be read, or that the service could not get
|
|
64
|
+
through, is rejected without being put back: requeueing it would be
|
|
65
|
+
asking for the same failure at once and for ever. Where it goes then is
|
|
66
|
+
the broker's business — a dead letter exchange if the queue has one,
|
|
67
|
+
nowhere if it has not.
|
|
68
|
+
"""
|
|
69
|
+
import json
|
|
70
|
+
|
|
71
|
+
try:
|
|
72
|
+
message = json.loads(body.decode())
|
|
73
|
+
except json.JSONDecodeError as exception:
|
|
74
|
+
self.io.error(
|
|
75
|
+
f"[{self.get_snake_short_class_name()}] Message is not JSON: {exception}",
|
|
76
|
+
fatal=False,
|
|
77
|
+
)
|
|
78
|
+
channel.basic_nack(method_frame.delivery_tag, requeue=False)
|
|
79
|
+
return
|
|
80
|
+
|
|
81
|
+
try:
|
|
82
|
+
self.process_message(message)
|
|
83
|
+
except Exception as exception:
|
|
84
|
+
import traceback
|
|
85
|
+
|
|
86
|
+
self.io.error(
|
|
87
|
+
f"[{self.get_snake_short_class_name()}] Message failed: {exception}",
|
|
88
|
+
fatal=False,
|
|
89
|
+
)
|
|
90
|
+
self.io.debug(traceback.format_exc())
|
|
91
|
+
channel.basic_nack(method_frame.delivery_tag, requeue=False)
|
|
92
|
+
return
|
|
93
|
+
|
|
94
|
+
channel.basic_ack(method_frame.delivery_tag)
|
|
95
|
+
|
|
96
|
+
def _run(self) -> None:
|
|
97
|
+
self.log(f"Consuming messages on queue « {self.queue_name} »...")
|
|
98
|
+
|
|
99
|
+
self.connector.consume(self.queue_name, self._on_message)
|
|
File without changes
|
wexample_queue-1.0.1/README.md
DELETED
|
@@ -1,143 +0,0 @@
|
|
|
1
|
-
# queue
|
|
2
|
-
|
|
3
|
-
Version: 1.0.1
|
|
4
|
-
|
|
5
|
-
The repository does not provide any concrete code that could be documented for now.
|
|
6
|
-
|
|
7
|
-
## Table of Contents
|
|
8
|
-
|
|
9
|
-
- [Installation](#installation)
|
|
10
|
-
- [Tests](#tests)
|
|
11
|
-
- [Integration in the Suite](#integration-in-the-suite)
|
|
12
|
-
- [Versioning & Compatibility Policy](#versioning--compatibility-policy)
|
|
13
|
-
- [License](#license)
|
|
14
|
-
- [About us](#about-us)
|
|
15
|
-
- [Known Limitations & Roadmap](#known-limitations--roadmap)
|
|
16
|
-
- [Status & Compatibility](#status--compatibility)
|
|
17
|
-
- [Useful Links](#useful-links)
|
|
18
|
-
- [Migration Notes](#migration-notes)
|
|
19
|
-
|
|
20
|
-
## Installation
|
|
21
|
-
|
|
22
|
-
```bash
|
|
23
|
-
pip install wexample-queue
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
Requires Python >=3.10.
|
|
27
|
-
|
|
28
|
-
The repository does not provide any concrete code that could be documented for now.
|
|
29
|
-
|
|
30
|
-
## Tests
|
|
31
|
-
|
|
32
|
-
This project uses `pytest` for testing and `pytest-cov` for code coverage analysis.
|
|
33
|
-
|
|
34
|
-
### Installation
|
|
35
|
-
|
|
36
|
-
First, install the required testing dependencies:
|
|
37
|
-
```bash
|
|
38
|
-
.venv/bin/python -m pip install pytest pytest-cov
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
### Basic Usage
|
|
42
|
-
|
|
43
|
-
Run all tests with coverage:
|
|
44
|
-
```bash
|
|
45
|
-
.venv/bin/python -m pytest --cov --cov-report=html
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
### Common Commands
|
|
49
|
-
```bash
|
|
50
|
-
# Run tests with coverage for a specific module
|
|
51
|
-
.venv/bin/python -m pytest --cov=your_module
|
|
52
|
-
|
|
53
|
-
# Show which lines are not covered
|
|
54
|
-
.venv/bin/python -m pytest --cov=your_module --cov-report=term-missing
|
|
55
|
-
|
|
56
|
-
# Generate an HTML coverage report
|
|
57
|
-
.venv/bin/python -m pytest --cov=your_module --cov-report=html
|
|
58
|
-
|
|
59
|
-
# Combine terminal and HTML reports
|
|
60
|
-
.venv/bin/python -m pytest --cov=your_module --cov-report=term-missing --cov-report=html
|
|
61
|
-
|
|
62
|
-
# Run specific test file with coverage
|
|
63
|
-
.venv/bin/python -m pytest tests/test_file.py --cov=your_module --cov-report=term-missing
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
### Viewing HTML Reports
|
|
67
|
-
|
|
68
|
-
After generating an HTML report, open `htmlcov/index.html` in your browser to view detailed line-by-line coverage information.
|
|
69
|
-
|
|
70
|
-
### Coverage Threshold
|
|
71
|
-
|
|
72
|
-
To enforce a minimum coverage percentage:
|
|
73
|
-
```bash
|
|
74
|
-
.venv/bin/python -m pytest --cov=your_module --cov-fail-under=80
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
This will cause the test suite to fail if coverage drops below 80%.
|
|
78
|
-
|
|
79
|
-
The repository does not provide any concrete code that could be documented for now.
|
|
80
|
-
|
|
81
|
-
## Integration in the Suite
|
|
82
|
-
|
|
83
|
-
This package is part of the Wexample Suite — a collection of high-quality, modular tools designed to work seamlessly together across multiple languages and environments.
|
|
84
|
-
|
|
85
|
-
### Related Packages
|
|
86
|
-
|
|
87
|
-
The suite includes packages for configuration management, file handling, prompts, and more. Each package can be used independently or as part of the integrated suite.
|
|
88
|
-
|
|
89
|
-
Visit the [Wexample Suite documentation](https://docs.wexample.com) for the complete package ecosystem.
|
|
90
|
-
|
|
91
|
-
## Versioning & Compatibility Policy
|
|
92
|
-
|
|
93
|
-
Wexample packages follow **Semantic Versioning** (SemVer):
|
|
94
|
-
|
|
95
|
-
- **MAJOR**: Breaking changes
|
|
96
|
-
- **MINOR**: New features, backward compatible
|
|
97
|
-
- **PATCH**: Bug fixes, backward compatible
|
|
98
|
-
|
|
99
|
-
We maintain backward compatibility within major versions and provide clear migration guides for breaking changes.
|
|
100
|
-
|
|
101
|
-
## License
|
|
102
|
-
|
|
103
|
-
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
|
104
|
-
|
|
105
|
-
Free to use in both personal and commercial projects.
|
|
106
|
-
|
|
107
|
-
## About us
|
|
108
|
-
|
|
109
|
-
[Wexample](https://wexample.com) stands as a cornerstone of the digital ecosystem — a collective of seasoned engineers, researchers, and creators driven by a relentless pursuit of technological excellence. More than a media platform, it has grown into a vibrant community where innovation meets craftsmanship, and where every line of code reflects a commitment to clarity, durability, and shared intelligence.
|
|
110
|
-
|
|
111
|
-
This packages suite embodies this spirit. Trusted by professionals and enthusiasts alike, it delivers a consistent, high-quality foundation for modern development — open, elegant, and battle-tested. Its reputation is built on years of collaboration, refinement, and rigorous attention to detail, making it a natural choice for those who demand both robustness and beauty in their tools.
|
|
112
|
-
|
|
113
|
-
Wexample cultivates a culture of mastery. Each package, each contribution carries the mark of a community that values precision, ethics, and innovation — a community proud to shape the future of digital craftsmanship.
|
|
114
|
-
|
|
115
|
-
## Known Limitations & Roadmap
|
|
116
|
-
|
|
117
|
-
Current limitations and planned features are tracked in the GitHub issues.
|
|
118
|
-
|
|
119
|
-
See the [project roadmap](https://github.com/wexample/python-queue/issues) for upcoming features and improvements.
|
|
120
|
-
|
|
121
|
-
## Status & Compatibility
|
|
122
|
-
|
|
123
|
-
**Maturity**: Production-ready
|
|
124
|
-
|
|
125
|
-
**Python Support**: >=3.10
|
|
126
|
-
|
|
127
|
-
**OS Support**: Linux, macOS, Windows
|
|
128
|
-
|
|
129
|
-
**Status**: Actively maintained
|
|
130
|
-
|
|
131
|
-
## Useful Links
|
|
132
|
-
|
|
133
|
-
- **Homepage**: https://github.com/wexample/python-queue
|
|
134
|
-
- **Documentation**: [docs.wexample.com](https://docs.wexample.com)
|
|
135
|
-
- **Issue Tracker**: https://github.com/wexample/python-queue/issues
|
|
136
|
-
- **Discussions**: https://github.com/wexample/python-queue/discussions
|
|
137
|
-
- **PyPI**: [pypi.org/project/wexample-queue](https://pypi.org/project/wexample-queue/)
|
|
138
|
-
|
|
139
|
-
## Migration Notes
|
|
140
|
-
|
|
141
|
-
When upgrading between major versions, refer to the migration guides in the documentation.
|
|
142
|
-
|
|
143
|
-
Breaking changes are clearly documented with upgrade paths and examples.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|