python-pq 0.5.2__py3-none-any.whl → 0.6.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
pq/__init__.py CHANGED
@@ -7,7 +7,7 @@ from pq.models import Periodic, Task, TaskStatus
7
7
  from pq.priority import Priority
8
8
  from pq.worker import PostExecuteHook, PreExecuteHook, TaskTimeoutError
9
9
 
10
- __version__ = "0.5.2"
10
+ __version__ = "0.6.0"
11
11
 
12
12
  __all__ = [
13
13
  "PQ",
@@ -0,0 +1,342 @@
1
+ Metadata-Version: 2.4
2
+ Name: python-pq
3
+ Version: 0.6.0
4
+ Summary: Postgres-backed job queue for Python
5
+ Keywords: postgres,job-queue,task-queue,background-jobs,worker
6
+ Author: ricwo
7
+ Author-email: ricwo <r@cogram.com>
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: OSI Approved :: MIT License
13
+ Classifier: Programming Language :: Python :: 3.13
14
+ Classifier: Programming Language :: Python :: 3.14
15
+ Classifier: Topic :: Software Development :: Libraries
16
+ Classifier: Topic :: System :: Distributed Computing
17
+ Classifier: Typing :: Typed
18
+ Requires-Dist: alembic>=1.17.2
19
+ Requires-Dist: click>=8.3.1
20
+ Requires-Dist: croniter>=6.0.0
21
+ Requires-Dist: dill>=0.4.0
22
+ Requires-Dist: loguru>=0.7.3
23
+ Requires-Dist: psycopg2-binary>=2.9.11
24
+ Requires-Dist: pydantic>=2.12.5
25
+ Requires-Dist: pydantic-settings>=2.12.0
26
+ Requires-Dist: sqlalchemy>=2.0.45
27
+ Requires-Dist: mkdocs-material>=9.7.1 ; extra == 'docs'
28
+ Requires-Dist: mkdocstrings[python]>=1.0.0 ; extra == 'docs'
29
+ Requires-Python: >=3.13, <3.15
30
+ Project-URL: Documentation, https://ricwo.github.io/pq/
31
+ Project-URL: Homepage, https://github.com/ricwo/pq
32
+ Project-URL: Issues, https://github.com/ricwo/pq/issues
33
+ Project-URL: Repository, https://github.com/ricwo/pq
34
+ Provides-Extra: docs
35
+ Description-Content-Type: text/markdown
36
+
37
+ # pq
38
+
39
+ [![PyPI](https://img.shields.io/pypi/v/python-pq)](https://pypi.org/project/python-pq/)
40
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
41
+ [![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
42
+
43
+ Postgres-backed job queue for Python with fork-based worker isolation.
44
+
45
+ If you already run Postgres, you don't need Redis or RabbitMQ to process background jobs. pq uses `SELECT ... FOR UPDATE SKIP LOCKED` to turn your existing database into a reliable task queue. Enqueue in the same transaction as your writes, and process tasks in isolated child processes that can't crash your worker.
46
+
47
+ ```python
48
+ from pq import PQ
49
+
50
+ pq = PQ("postgresql://localhost/mydb")
51
+ pq.run_db_migrations()
52
+
53
+ def send_email(to: str, subject: str) -> None:
54
+ ...
55
+
56
+ pq.enqueue(send_email, to="user@example.com", subject="Hello")
57
+ pq.run_worker()
58
+ ```
59
+
60
+ ## Install
61
+
62
+ ```bash
63
+ uv add python-pq
64
+ ```
65
+
66
+ Or with pip:
67
+
68
+ ```bash
69
+ pip install python-pq
70
+ ```
71
+
72
+ Requires PostgreSQL and Python 3.13+.
73
+
74
+ ## Features
75
+
76
+ - **Fork isolation** -- Each task runs in a forked child process. If it OOMs, segfaults, or crashes, the worker keeps running.
77
+ - **No extra infrastructure** -- Uses your existing Postgres. No broker to deploy, monitor, or lose data.
78
+ - **Transactional enqueueing** -- Enqueue tasks in the same database transaction as your writes. If the transaction rolls back, the task is never created.
79
+ - **Periodic tasks** -- Schedule with intervals (`timedelta`) or cron expressions. Control overlap, pause/resume without deleting.
80
+ - **Priority queues** -- Five levels from `BATCH` (0) to `CRITICAL` (100). Dedicate workers to specific priority tiers.
81
+ - **Lifecycle hooks** -- Run `pre_execute` / `post_execute` code in the forked child, safe for fork-unsafe libraries like OpenTelemetry.
82
+
83
+ ## Tasks
84
+
85
+ ### Enqueueing
86
+
87
+ Pass any importable function with its arguments:
88
+
89
+ ```python
90
+ def greet(name: str) -> None:
91
+ print(f"Hello, {name}!")
92
+
93
+ pq.enqueue(greet, name="World")
94
+ pq.enqueue(greet, "World") # Positional args work too
95
+ ```
96
+
97
+ ### Delayed execution
98
+
99
+ ```python
100
+ from datetime import datetime, timedelta, UTC
101
+
102
+ pq.enqueue(greet, "World", run_at=datetime.now(UTC) + timedelta(hours=1))
103
+ ```
104
+
105
+ ### Priority
106
+
107
+ ```python
108
+ from pq import Priority
109
+
110
+ pq.enqueue(task, priority=Priority.CRITICAL) # 100 - runs first
111
+ pq.enqueue(task, priority=Priority.HIGH) # 75
112
+ pq.enqueue(task, priority=Priority.NORMAL) # 50 (default)
113
+ pq.enqueue(task, priority=Priority.LOW) # 25
114
+ pq.enqueue(task, priority=Priority.BATCH) # 0 - runs last
115
+ ```
116
+
117
+ ### Cancellation
118
+
119
+ ```python
120
+ task_id = pq.enqueue(my_task)
121
+ pq.cancel(task_id)
122
+ ```
123
+
124
+ ### Client IDs
125
+
126
+ Use `client_id` for idempotency and lookups:
127
+
128
+ ```python
129
+ pq.enqueue(process_order, order_id=123, client_id="order-123")
130
+
131
+ task = pq.get_task_by_client_id("order-123")
132
+ # Duplicate client_id raises IntegrityError
133
+ ```
134
+
135
+ ### Upsert
136
+
137
+ Insert or update a task by `client_id`. Useful for debouncing -- only the latest version runs:
138
+
139
+ ```python
140
+ pq.upsert(send_email, to="a@b.com", client_id="welcome-email")
141
+
142
+ # Second call updates the existing task (resets to PENDING)
143
+ pq.upsert(send_email, to="new@b.com", client_id="welcome-email")
144
+ ```
145
+
146
+ ## Periodic Tasks
147
+
148
+ ### Intervals
149
+
150
+ ```python
151
+ from datetime import timedelta
152
+
153
+ def heartbeat() -> None:
154
+ print("alive")
155
+
156
+ pq.schedule(heartbeat, run_every=timedelta(minutes=5))
157
+ ```
158
+
159
+ ### Cron expressions
160
+
161
+ ```python
162
+ pq.schedule(weekly_report, cron="0 9 * * 1") # Monday 9am
163
+ ```
164
+
165
+ ### With arguments
166
+
167
+ ```python
168
+ pq.schedule(report, run_every=timedelta(hours=1), report_type="hourly")
169
+ ```
170
+
171
+ ### Overlap control
172
+
173
+ By default, periodic tasks don't overlap -- if an instance is still running when the next tick arrives, the tick is skipped:
174
+
175
+ ```python
176
+ # Default: max_concurrent=1, no overlap
177
+ pq.schedule(sync_inventory, run_every=timedelta(minutes=5))
178
+
179
+ # Allow unlimited concurrency
180
+ pq.schedule(fast_task, run_every=timedelta(seconds=30), max_concurrent=None)
181
+ ```
182
+
183
+ ### Pausing and resuming
184
+
185
+ ```python
186
+ # Pause -- task stays in the database but won't run
187
+ pq.schedule(sync_inventory, run_every=timedelta(minutes=5), active=False)
188
+
189
+ # Resume
190
+ pq.schedule(sync_inventory, run_every=timedelta(minutes=5), active=True)
191
+ ```
192
+
193
+ ### Multiple schedules
194
+
195
+ Use `key` to register the same function with different configurations:
196
+
197
+ ```python
198
+ pq.schedule(sync_data, run_every=timedelta(hours=1), key="us", region="us")
199
+ pq.schedule(sync_data, run_every=timedelta(hours=2), key="eu", region="eu")
200
+
201
+ pq.unschedule(sync_data, key="us")
202
+ ```
203
+
204
+ ## Workers
205
+
206
+ ### Running
207
+
208
+ ```python
209
+ pq.run_worker(poll_interval=1.0) # Run forever
210
+ processed = pq.run_worker_once() # Process single task (for testing)
211
+ ```
212
+
213
+ ### Timeout
214
+
215
+ Kill tasks that run too long:
216
+
217
+ ```python
218
+ pq.run_worker(max_runtime=300) # 5 minute timeout per task
219
+ ```
220
+
221
+ ### Priority-dedicated workers
222
+
223
+ Reserve workers for high-priority tasks:
224
+
225
+ ```python
226
+ from pq import Priority
227
+
228
+ # This worker only processes CRITICAL and HIGH
229
+ pq.run_worker(priorities={Priority.CRITICAL, Priority.HIGH})
230
+ ```
231
+
232
+ ### Lifecycle hooks
233
+
234
+ Run code before/after each task in the forked child process:
235
+
236
+ ```python
237
+ from pq import PQ, Task, Periodic
238
+
239
+ def setup_tracing(task: Task | Periodic) -> None:
240
+ print(f"Starting: {task.name}")
241
+
242
+ def flush_tracing(task: Task | Periodic, error: Exception | None) -> None:
243
+ if error:
244
+ print(f"Failed: {error}")
245
+
246
+ pq.run_worker(pre_execute=setup_tracing, post_execute=flush_tracing)
247
+ ```
248
+
249
+ Hooks run in the forked child, making them safe for fork-unsafe resources like OpenTelemetry.
250
+
251
+ ## Serialization
252
+
253
+ Arguments are serialized automatically:
254
+
255
+ | Type | Method |
256
+ |------|--------|
257
+ | JSON-serializable (str, int, list, dict) | JSON |
258
+ | Pydantic models | `model_dump()` → JSON |
259
+ | Custom objects, lambdas | dill (pickle) |
260
+
261
+ ## Async tasks
262
+
263
+ Async handlers work without any changes:
264
+
265
+ ```python
266
+ import httpx
267
+
268
+ async def fetch(url: str) -> None:
269
+ async with httpx.AsyncClient() as client:
270
+ response = await client.get(url)
271
+ print(response.status_code)
272
+
273
+ pq.enqueue(fetch, "https://example.com")
274
+ ```
275
+
276
+ ## Error handling
277
+
278
+ Failed tasks are marked with status `FAILED` and the error is stored:
279
+
280
+ ```python
281
+ for task in pq.list_failed():
282
+ print(f"{task.name}: {task.error}")
283
+
284
+ pq.clear_failed(before=datetime.now(UTC) - timedelta(days=7))
285
+ pq.clear_completed(before=datetime.now(UTC) - timedelta(days=1))
286
+ ```
287
+
288
+ ## How it works
289
+
290
+ Every task runs in a forked child process:
291
+
292
+ ```
293
+ Worker (parent)
294
+ |
295
+ +-- fork() -> Child executes task -> exits
296
+ | (OOM/crash only affects child)
297
+ |
298
+ +-- Continues processing next task
299
+ ```
300
+
301
+ The parent monitors via `os.wait4()` and detects timeout, OOM (SIGKILL), and signal-based crashes. The child process exits after every task, giving you true memory isolation.
302
+
303
+ Multiple workers can run in parallel. Tasks are claimed atomically with PostgreSQL's `FOR UPDATE SKIP LOCKED`, so each task runs exactly once.
304
+
305
+ ## Alternatives
306
+
307
+ There are good options in this space. pq makes different tradeoffs:
308
+
309
+ | | Broker | Isolation | Use case |
310
+ |---|---|---|---|
311
+ | **pq** | Postgres | Fork (process-per-task) | Teams already on Postgres who want fewer moving parts |
312
+ | **Celery** | Redis/RabbitMQ | Per-worker process | Large-scale, multi-language, established teams |
313
+ | **RQ** | Redis | Per-worker process | Simple Redis-based queues |
314
+ | **Dramatiq** | Redis/RabbitMQ | Per-worker process/thread | Celery alternative with better defaults |
315
+ | **ARQ** | Redis | Async (single process) | Async-first applications |
316
+ | **Procrastinate** | Postgres | Async (single process) | Async-first, Postgres-backed, Django integration |
317
+
318
+ pq is a good fit when:
319
+ - You already run Postgres and don't want to add Redis or RabbitMQ
320
+ - You want transactional enqueueing (enqueue atomically with your writes)
321
+ - You need true process isolation per task (OOM/crash safety)
322
+ - You want periodic tasks with overlap control, pause/resume, and cron
323
+
324
+ pq is not the right choice when:
325
+ - You need very high throughput (10,000+ jobs/second) -- use a dedicated broker
326
+ - You need cross-language workers -- Celery or a dedicated queue service is better
327
+ - You need complex workflows (DAGs, chaining, fan-out) -- look at Temporal or Prefect
328
+
329
+ ## Documentation
330
+
331
+ Full docs at [ricwo.github.io/pq](https://ricwo.github.io/pq/).
332
+
333
+ ## Development
334
+
335
+ ```bash
336
+ make dev # Start Postgres
337
+ uv run pytest # Run tests
338
+ ```
339
+
340
+ ## License
341
+
342
+ MIT
@@ -1,4 +1,4 @@
1
- pq/__init__.py,sha256=0KAItMJA7i6_Ki3natQiH7VfVtYvW6hixxGt9ggD5R8,461
1
+ pq/__init__.py,sha256=A2IkhxWaIDyNK9Xp-O_XRsHbp09HgqEOrRS4uOGfk10,461
2
2
  pq/client.py,sha256=f9x_ESSq_d0OTye29reo6Ve0fOMXMROhjzTGt5KIGf8,18787
3
3
  pq/config.py,sha256=U7dW-LVHy4feLUDQAmxKSF8bavtwxMY4thYCNMR7nDc,293
4
4
  pq/logging.py,sha256=nEEMHQkL5s84wxylhsb1-4DqmAUQGALUtKG5elSFcwA,644
@@ -17,7 +17,8 @@ pq/priority.py,sha256=edqpQeGiJrxcmpJ5MZCvwTD9F6sIZbRoI1mx_LmUxAs,264
17
17
  pq/registry.py,sha256=1RFfB-dXXM2qKwLF6XeYeUm9buca1_fQvKQDnpfsXKs,2306
18
18
  pq/serialization.py,sha256=ZspG9qO4YYi-xCrtPGCg8yrzqRI2CGYnO_Ewjh8lE1E,3291
19
19
  pq/worker.py,sha256=uBbsZAOhhjOCb2CG-32QlRIRJO51TNu8e9AF2BATAmI,20418
20
- python_pq-0.5.2.dist-info/WHEEL,sha256=eh7sammvW2TypMMMGKgsM83HyA_3qQ5Lgg3ynoecH3M,79
21
- python_pq-0.5.2.dist-info/entry_points.txt,sha256=fpK4uA3gQaFpcDmdy5GFbP4VP7gaJcn4H4GJyMzRIM8,36
22
- python_pq-0.5.2.dist-info/METADATA,sha256=NCfV3SHaWhUFJZzOZE_r187D8idjJ_i5wXG6-fXVDWk,8134
23
- python_pq-0.5.2.dist-info/RECORD,,
20
+ python_pq-0.6.0.dist-info/licenses/LICENSE,sha256=Bx9dUtjCpaluWGalMqkI3SGkpPxx_AYdJdmV5ptCM4g,1062
21
+ python_pq-0.6.0.dist-info/WHEEL,sha256=eh7sammvW2TypMMMGKgsM83HyA_3qQ5Lgg3ynoecH3M,79
22
+ python_pq-0.6.0.dist-info/entry_points.txt,sha256=fpK4uA3gQaFpcDmdy5GFbP4VP7gaJcn4H4GJyMzRIM8,36
23
+ python_pq-0.6.0.dist-info/METADATA,sha256=UYexSkZiw8BzB9mmSBWTaIBFwHwqlFUg_UWbsx4jpMI,9714
24
+ python_pq-0.6.0.dist-info/RECORD,,
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 ricwo
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -1,351 +0,0 @@
1
- Metadata-Version: 2.3
2
- Name: python-pq
3
- Version: 0.5.2
4
- Summary: Postgres-backed job queue for Python
5
- Author: ricwo
6
- Author-email: ricwo <r@cogram.com>
7
- Requires-Dist: alembic>=1.17.2
8
- Requires-Dist: click>=8.3.1
9
- Requires-Dist: croniter>=6.0.0
10
- Requires-Dist: dill>=0.4.0
11
- Requires-Dist: loguru>=0.7.3
12
- Requires-Dist: psycopg2-binary>=2.9.11
13
- Requires-Dist: pydantic>=2.12.5
14
- Requires-Dist: pydantic-settings>=2.12.0
15
- Requires-Dist: sqlalchemy>=2.0.45
16
- Requires-Dist: mkdocs-material>=9.7.1 ; extra == 'docs'
17
- Requires-Dist: mkdocstrings[python]>=1.0.0 ; extra == 'docs'
18
- Requires-Python: >=3.13, <3.15
19
- Provides-Extra: docs
20
- Description-Content-Type: text/markdown
21
-
22
- # pq
23
-
24
- Postgres-backed job queue for Python with fork-based worker isolation.
25
-
26
- ## Features
27
-
28
- - **Fork isolation** - Each task runs in a forked process. OOM or crashes don't affect the worker.
29
- - **Natural Python API** - Pass functions directly with `*args, **kwargs`.
30
- - **Periodic tasks** - Schedule with intervals or cron expressions.
31
- - **Priority queues** - Five levels, higher priority tasks run first.
32
- - **Async support** - Async handlers work seamlessly.
33
- - **Concurrent workers** - `FOR UPDATE SKIP LOCKED` prevents duplicate processing.
34
-
35
- ## Installation
36
-
37
- ```bash
38
- uv add python-pq
39
- ```
40
-
41
- Requires PostgreSQL and Python 3.13+.
42
-
43
- ## Quick Start
44
-
45
- ```python
46
- from pq import PQ
47
-
48
- pq = PQ("postgresql://localhost/mydb")
49
- pq.run_db_migrations() # Creates/updates tables
50
-
51
- def send_email(to: str, subject: str) -> None:
52
- print(f"Sending to {to}: {subject}")
53
-
54
- pq.enqueue(send_email, to="user@example.com", subject="Hello")
55
- pq.run_worker()
56
- ```
57
-
58
- ## Tasks
59
-
60
- ### Enqueueing
61
-
62
- ```python
63
- def greet(name: str) -> None:
64
- print(f"Hello, {name}!")
65
-
66
- pq.enqueue(greet, name="World")
67
- pq.enqueue(greet, "World") # Positional args work too
68
- ```
69
-
70
- ### Delayed Execution
71
-
72
- ```python
73
- from datetime import datetime, timedelta, UTC
74
-
75
- pq.enqueue(greet, "World", run_at=datetime.now(UTC) + timedelta(hours=1))
76
- ```
77
-
78
- ### Priority
79
-
80
- ```python
81
- from pq import Priority
82
-
83
- pq.enqueue(task, priority=Priority.CRITICAL) # 100 - highest
84
- pq.enqueue(task, priority=Priority.HIGH) # 75
85
- pq.enqueue(task, priority=Priority.NORMAL) # 50 (default)
86
- pq.enqueue(task, priority=Priority.LOW) # 25
87
- pq.enqueue(task, priority=Priority.BATCH) # 0 - lowest
88
- ```
89
-
90
- ### Cancellation
91
-
92
- ```python
93
- task_id = pq.enqueue(my_task)
94
- pq.cancel(task_id) # Returns True if found and cancelled
95
- ```
96
-
97
- ### Client IDs
98
-
99
- Use `client_id` for idempotency and lookups:
100
-
101
- ```python
102
- # Enqueue with a client-provided ID
103
- pq.enqueue(process_order, order_id=123, client_id="order-123")
104
-
105
- # Look up by client_id
106
- task = pq.get_task_by_client_id("order-123")
107
- if task:
108
- pq.cancel(task.id)
109
-
110
- # Duplicate client_id raises IntegrityError
111
- ```
112
-
113
- ### Upsert
114
-
115
- Insert or update a task by `client_id`:
116
-
117
- ```python
118
- # First call creates task
119
- pq.upsert(send_email, to="a@b.com", client_id="welcome-email")
120
-
121
- # Second call updates the existing task (resets to PENDING)
122
- pq.upsert(send_email, to="new@b.com", client_id="welcome-email")
123
- ```
124
-
125
- Unlike `enqueue()`, `upsert()` requires `client_id` and won't raise on duplicates.
126
-
127
- ## Periodic Tasks
128
-
129
- ### Intervals
130
-
131
- ```python
132
- from datetime import timedelta
133
-
134
- def heartbeat() -> None:
135
- print("alive")
136
-
137
- pq.schedule(heartbeat, run_every=timedelta(minutes=5))
138
- ```
139
-
140
- ### Cron Expressions
141
-
142
- ```python
143
- def weekly_report() -> None:
144
- print("generating report...")
145
-
146
- pq.schedule(weekly_report, cron="0 9 * * 1") # Monday 9am
147
- ```
148
-
149
- ### With Arguments
150
-
151
- ```python
152
- pq.schedule(report, run_every=timedelta(hours=1), report_type="hourly")
153
- ```
154
-
155
- ### Overlap Control
156
-
157
- By default, periodic tasks don't overlap — if an instance is still running when the next tick arrives, the tick is skipped:
158
-
159
- ```python
160
- # Default: max_concurrent=1, no overlap
161
- pq.schedule(sync_inventory, run_every=timedelta(minutes=5))
162
-
163
- # Opt in to unlimited concurrency (overlap allowed)
164
- pq.schedule(fast_idempotent_task, run_every=timedelta(seconds=30), max_concurrent=None)
165
- ```
166
-
167
- The lock auto-expires after `max_runtime` seconds (or 1 hour by default) for crash safety.
168
-
169
- ### Pausing & Resuming
170
-
171
- Disable a periodic task without removing it:
172
-
173
- ```python
174
- # Pause - task stays in the database but won't run
175
- pq.schedule(sync_inventory, run_every=timedelta(minutes=5), active=False)
176
-
177
- # Resume
178
- pq.schedule(sync_inventory, run_every=timedelta(minutes=5), active=True)
179
- ```
180
-
181
- New schedules are active by default.
182
-
183
- ### Multiple Schedules (Key)
184
-
185
- Use `key` to register the same function multiple times with different configurations:
186
-
187
- ```python
188
- pq.schedule(sync_data, run_every=timedelta(hours=1), key="us", region="us")
189
- pq.schedule(sync_data, run_every=timedelta(hours=2), key="eu", region="eu")
190
-
191
- # Unschedule only the US schedule
192
- pq.unschedule(sync_data, key="us")
193
- ```
194
-
195
- Omitting `key` defaults to `""` — backward-compatible with single-schedule usage.
196
-
197
- ### Unscheduling
198
-
199
- ```python
200
- pq.unschedule(heartbeat) # Returns True if found
201
- ```
202
-
203
- ## Workers
204
-
205
- ### Running
206
-
207
- ```python
208
- # Run forever, poll every second when idle
209
- pq.run_worker(poll_interval=1.0)
210
-
211
- # Process single task (for testing)
212
- processed = pq.run_worker_once()
213
- ```
214
-
215
- ### Timeout
216
-
217
- Kill tasks that run too long:
218
-
219
- ```python
220
- pq.run_worker(max_runtime=300) # 5 minute timeout
221
- ```
222
-
223
- ### Priority-Dedicated Workers
224
-
225
- Reserve workers for high-priority tasks:
226
-
227
- ```python
228
- from pq import Priority
229
-
230
- # This worker only processes CRITICAL and HIGH
231
- pq.run_worker(priorities={Priority.CRITICAL, Priority.HIGH})
232
- ```
233
-
234
- Run multiple workers in separate terminals:
235
-
236
- ```bash
237
- # Terminal 1: High-priority only
238
- python -c "from myapp import pq; from pq import Priority; pq.run_worker(priorities={Priority.CRITICAL, Priority.HIGH})"
239
-
240
- # Terminal 2-3: All priorities
241
- python -c "from myapp import pq; pq.run_worker()"
242
- ```
243
-
244
- ### Lifecycle Hooks
245
-
246
- Run code before/after task execution in the forked child process:
247
-
248
- ```python
249
- from pq import PQ, Task, Periodic
250
-
251
- def setup_tracing(task: Task | Periodic) -> None:
252
- # Initialize OTel tracer, create span, etc.
253
- print(f"Starting task: {task.name}")
254
-
255
- def flush_tracing(task: Task | Periodic, error: Exception | None) -> None:
256
- # Flush traces, cleanup resources
257
- if error:
258
- print(f"Task failed: {error}")
259
-
260
- pq.run_worker(pre_execute=setup_tracing, post_execute=flush_tracing)
261
- ```
262
-
263
- Hooks run in the forked child, making them safe for fork-unsafe resources like OpenTelemetry.
264
-
265
- ## Serialization
266
-
267
- Arguments are serialized automatically:
268
-
269
- | Type | Method |
270
- |------|--------|
271
- | JSON-serializable (str, int, list, dict) | JSON |
272
- | Pydantic models | `model_dump()` → JSON |
273
- | Custom objects, lambdas | dill (pickle) |
274
-
275
- ```python
276
- from typing import Callable
277
- from pydantic import BaseModel
278
-
279
- class User(BaseModel):
280
- id: int
281
- email: str
282
-
283
- def process(user: User, transform: Callable[[int], int]) -> None:
284
- print(transform(user.id))
285
-
286
- pq.enqueue(process, User(id=1, email="a@b.com"), transform=lambda x: x * 2)
287
- ```
288
-
289
- ## Async Tasks
290
-
291
- ```python
292
- import httpx
293
-
294
- async def fetch(url: str) -> None:
295
- async with httpx.AsyncClient() as client:
296
- response = await client.get(url)
297
- print(response.status_code)
298
-
299
- pq.enqueue(fetch, "https://example.com")
300
- ```
301
-
302
- ## Error Handling
303
-
304
- Failed tasks are marked with status `FAILED`:
305
-
306
- ```python
307
- for task in pq.list_failed():
308
- print(f"{task.name}: {task.error}")
309
-
310
- # Cleanup
311
- pq.clear_failed(before=datetime.now(UTC) - timedelta(days=7))
312
- pq.clear_completed(before=datetime.now(UTC) - timedelta(days=1))
313
- ```
314
-
315
- ## Architecture
316
-
317
- ### Fork Isolation
318
-
319
- Every task runs in a forked child process:
320
-
321
- ```
322
- Worker (parent)
323
- │
324
- ├── fork() → Child executes task → exits
325
- │ (OOM/crash only affects child)
326
- │
327
- └── Continues processing next task
328
- ```
329
-
330
- The parent monitors via `os.wait4()` and detects:
331
-
332
- - **Timeout** - Task exceeded `max_runtime`
333
- - **OOM** - Killed by SIGKILL (OOM killer)
334
- - **Signals** - Killed by other signals
335
-
336
- ### Concurrent Workers
337
-
338
- Multiple workers can run in parallel. Tasks are claimed with `FOR UPDATE SKIP LOCKED`, ensuring each task runs exactly once.
339
-
340
- ## Development
341
-
342
- ```bash
343
- make dev # Start Postgres
344
- uv run pytest # Run tests
345
- ```
346
-
347
- See [CLAUDE.md](CLAUDE.md) for full development instructions.
348
-
349
- ## License
350
-
351
- MIT