taskscheduler-client 0.6.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (26) hide show
  1. taskscheduler_client-0.6.0/.gitignore +15 -0
  2. taskscheduler_client-0.6.0/PKG-INFO +325 -0
  3. taskscheduler_client-0.6.0/README.md +297 -0
  4. taskscheduler_client-0.6.0/examples/demo.py +170 -0
  5. taskscheduler_client-0.6.0/pyproject.toml +64 -0
  6. taskscheduler_client-0.6.0/scripts/apply_migrations.py +103 -0
  7. taskscheduler_client-0.6.0/src/taskscheduler/__init__.py +141 -0
  8. taskscheduler_client-0.6.0/src/taskscheduler/config.py +190 -0
  9. taskscheduler_client-0.6.0/src/taskscheduler/context.py +185 -0
  10. taskscheduler_client-0.6.0/src/taskscheduler/cron.py +76 -0
  11. taskscheduler_client-0.6.0/src/taskscheduler/errors.py +56 -0
  12. taskscheduler_client-0.6.0/src/taskscheduler/events.py +159 -0
  13. taskscheduler_client-0.6.0/src/taskscheduler/job.py +207 -0
  14. taskscheduler_client-0.6.0/src/taskscheduler/models.py +119 -0
  15. taskscheduler_client-0.6.0/src/taskscheduler/retry.py +108 -0
  16. taskscheduler_client-0.6.0/src/taskscheduler/scheduler.py +558 -0
  17. taskscheduler_client-0.6.0/src/taskscheduler/serde.py +260 -0
  18. taskscheduler_client-0.6.0/src/taskscheduler/storage.py +854 -0
  19. taskscheduler_client-0.6.0/src/taskscheduler/transport.py +178 -0
  20. taskscheduler_client-0.6.0/src/taskscheduler/worker.py +551 -0
  21. taskscheduler_client-0.6.0/tests/integration/conftest.py +194 -0
  22. taskscheduler_client-0.6.0/tests/integration/test_scheduler.py +324 -0
  23. taskscheduler_client-0.6.0/tests/integration/test_worker.py +503 -0
  24. taskscheduler_client-0.6.0/tests/unit/test_config_and_registry.py +230 -0
  25. taskscheduler_client-0.6.0/tests/unit/test_cron_and_retry.py +118 -0
  26. taskscheduler_client-0.6.0/tests/unit/test_serde.py +148 -0
@@ -0,0 +1,15 @@
1
+ # Python build + tooling artefacts. The Gradle-oriented root .gitignore doesn't cover these.
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ build/
6
+ dist/
7
+
8
+ .venv/
9
+ venv/
10
+
11
+ .mypy_cache/
12
+ .pytest_cache/
13
+ .ruff_cache/
14
+ .coverage
15
+ htmlcov/
@@ -0,0 +1,325 @@
1
+ Metadata-Version: 2.4
2
+ Name: taskscheduler-client
3
+ Version: 0.6.0
4
+ Summary: Async Python SDK for TaskScheduler — durable distributed background jobs on PostgreSQL + RabbitMQ
5
+ Project-URL: Homepage, https://github.com/4aK-Boris/TaskScheduler
6
+ Project-URL: Documentation, https://github.com/4aK-Boris/TaskScheduler/blob/master/clients/python/README.md
7
+ Author: TaskScheduler
8
+ License: Apache-2.0
9
+ Keywords: asyncio,background,jobs,postgresql,rabbitmq,scheduler
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Framework :: AsyncIO
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Programming Language :: Python :: 3.10
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Classifier: Topic :: System :: Distributed Computing
18
+ Requires-Python: >=3.10
19
+ Requires-Dist: aio-pika>=9.5
20
+ Requires-Dist: croniter>=6.0
21
+ Requires-Dist: psycopg[binary,pool]>=3.2
22
+ Provides-Extra: dev
23
+ Requires-Dist: mypy>=1.14; extra == 'dev'
24
+ Requires-Dist: pytest-asyncio>=0.25; extra == 'dev'
25
+ Requires-Dist: pytest>=8.3; extra == 'dev'
26
+ Requires-Dist: ruff>=0.9; extra == 'dev'
27
+ Description-Content-Type: text/markdown
28
+
29
+ # TaskScheduler — Python SDK
30
+
31
+ Async Python client for [TaskScheduler](https://github.com/4aK-Boris/TaskScheduler). A Python service becomes a
32
+ first-class participant in the same job system as your Kotlin services: it enqueues work,
33
+ runs handlers, and shows up on the same dashboard with the same retry, cancellation and
34
+ progress semantics.
35
+
36
+ ```python
37
+ from dataclasses import dataclass
38
+ from taskscheduler import Scheduler, SchedulerConfig, job_type
39
+
40
+ @job_type
41
+ @dataclass
42
+ class SendInvoice:
43
+ order_id: int
44
+
45
+ async with Scheduler(SchedulerConfig(dsn="postgresql://scheduler:scheduler@localhost/scheduler")) as s:
46
+ await s.enqueue(SendInvoice(order_id=42))
47
+ ```
48
+
49
+ ## How it fits together
50
+
51
+ The SDK speaks the same wire protocol as the Kotlin client — it is not a proxy in front of
52
+ it. PostgreSQL holds all job state; RabbitMQ carries nothing but a 16-byte job id as a
53
+ delivery hint.
54
+
55
+ ```
56
+ ┌──────────────────────────┐ ┌──────────────────────────┐
57
+ │ scheduler-infra (Kotlin) │ │ your Python service │
58
+ │ • owns the schema │ │ • Scheduler.enqueue() │
59
+ │ • outbox → RabbitMQ │ PG + │ • WorkerPool runs jobs │
60
+ │ • recurring cron │ Rabbit │ • heartbeats its lease │
61
+ │ • orphan recovery │◄──────►│ │
62
+ │ • dashboard :8080 │ │ │
63
+ └──────────────────────────┘ └──────────────────────────┘
64
+ └────────► PostgreSQL ◄────────┘
65
+ └────────► RabbitMQ ◄────────┘
66
+ ```
67
+
68
+ **`scheduler-infra` is required.** It owns the Flyway migrations and the background loops
69
+ that neither client implements: publishing the outbox to RabbitMQ, firing cron definitions,
70
+ recovering jobs whose worker died, and retention. This SDK verifies the schema version at
71
+ startup and refuses to run against a database that is too old.
72
+
73
+ ## Install
74
+
75
+ ```bash
76
+ pip install taskscheduler-client # or: uv pip install taskscheduler-client
77
+ ```
78
+
79
+ Requires Python 3.10+, and a RabbitMQ with the `rabbitmq_delayed_message_exchange` plugin
80
+ enabled (the same requirement the Kotlin side has — it is how delays and retry backoff work).
81
+
82
+ **Versioning.** The client shares one version with the rest of the project:
83
+ `taskscheduler-client 0.6.0` is the client for `scheduler-infra 0.6.0`, and the two are
84
+ released together. Run a client older than your infra and it may not know about a newer
85
+ column; newer, and it fails fast on the schema check. CI enforces that
86
+ `pyproject.toml`, `taskscheduler.__version__` and Gradle's `schedulerVersion` agree.
87
+
88
+ **On Windows**, select the other event loop before starting anything — psycopg cannot run on
89
+ the `ProactorEventLoop` that asyncio uses by default there:
90
+
91
+ ```python
92
+ if sys.platform == "win32":
93
+ asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
94
+ ```
95
+
96
+ The SDK raises a `ConfigurationError` naming this if you forget, rather than hanging on a
97
+ connection-pool timeout.
98
+
99
+ ## Producing jobs
100
+
101
+ A payload is a plain dataclass. `@job_type` gives it a stable name that is stored in
102
+ `job.payload_type`:
103
+
104
+ ```python
105
+ @job_type # -> "billing.jobs.SendInvoice"
106
+ @dataclass
107
+ class SendInvoice:
108
+ order_id: int
109
+ dry_run: bool = False
110
+
111
+ @job_type("billing.SendInvoice.v2") # pin it before renaming or moving the class
112
+ @dataclass
113
+ class SendInvoiceV2:
114
+ ...
115
+ ```
116
+
117
+ ```python
118
+ # now
119
+ await scheduler.enqueue(SendInvoice(order_id=42), queue="billing", priority=7)
120
+
121
+ # at a moment
122
+ await scheduler.schedule_at(SendReminder(order_id=42), at=datetime(2026, 6, 1, 10, tzinfo=timezone.utc))
123
+ await scheduler.schedule_in(SendReminder(order_id=42), delay=timedelta(days=1))
124
+
125
+ # at most one active job per key
126
+ await scheduler.enqueue_once(f"sync-user-{user_id}", SyncUser(user_id))
127
+
128
+ # strictly in order
129
+ await scheduler.chain(ExtractData(), TransformData(), LoadData())
130
+
131
+ # after a fan-out finishes
132
+ a = await scheduler.enqueue(LoadProductCache())
133
+ b = await scheduler.enqueue(LoadUserCache())
134
+ await scheduler.enqueue_after(StartPricingEngine(), wait_for=[a, b])
135
+
136
+ # on a cron
137
+ await scheduler.recurring("nightly-rollup", "0 3 * * *", NightlyRollup(), timezone_name="Europe/Berlin")
138
+ ```
139
+
140
+ Every enqueue writes the job row and its outbox row in **one transaction**, so a job
141
+ becomes visible only if your surrounding business transaction commits.
142
+
143
+ ## Consuming jobs
144
+
145
+ ```python
146
+ from taskscheduler import HandlerRegistry, JobContext, RabbitConfig, WorkerConfig, WorkerPool
147
+
148
+ registry = HandlerRegistry()
149
+
150
+ @registry.handler(SendInvoice, retry_policy=ExponentialBackoff(max_attempts=5))
151
+ async def send_invoice(ctx: JobContext, job: SendInvoice) -> None:
152
+ await billing.send(job.order_id, idempotency_key=str(ctx.job_id))
153
+
154
+ worker = WorkerPool(
155
+ scheduler_config=SchedulerConfig(dsn=DSN, node_id="billing-1"),
156
+ worker_config=WorkerConfig(node_id="billing-1").queue("billing", concurrency=8),
157
+ rabbit_config=RabbitConfig(url=AMQP_URL, queues=["billing"]),
158
+ registry=registry,
159
+ )
160
+
161
+ async with worker:
162
+ await asyncio.Event().wait() # run until the process is stopped
163
+ ```
164
+
165
+ Class-based handlers work too, when you need `on_final_failure` or dependency injection:
166
+
167
+ ```python
168
+ class SendInvoiceHandler(JobHandler[SendInvoice]):
169
+ payload = SendInvoice
170
+ retry_policy = ExponentialBackoff(max_attempts=5)
171
+
172
+ def __init__(self, billing: BillingClient) -> None:
173
+ self._billing = billing
174
+
175
+ async def execute(self, ctx: JobContext, job: SendInvoice) -> None:
176
+ await self._billing.send(job.order_id)
177
+
178
+ async def on_final_failure(self, ctx: JobContext, job: SendInvoice, error: BaseException) -> None:
179
+ await alerts.page(f"invoice {job.order_id} failed permanently")
180
+
181
+ registry.register(SendInvoiceHandler(billing))
182
+ ```
183
+
184
+ ### Progress and cancellation
185
+
186
+ ```python
187
+ @registry.handler(ReindexCatalog)
188
+ async def reindex(ctx: JobContext, job: ReindexCatalog) -> None:
189
+ bar = ctx.progress_bar(len(job.product_ids))
190
+ for product_id in job.product_ids:
191
+ if await ctx.is_cancellation_requested():
192
+ raise JobCancellationError() # ends as CANCELLED, not FAILED
193
+ await index(product_id)
194
+ await bar.succeeded()
195
+ ```
196
+
197
+ Progress writes are throttled to one per second, so calling them in a tight loop is fine.
198
+ A cancelled job that ignores the flag is cancelled outright after
199
+ `WorkerConfig.cancel_grace_seconds`.
200
+
201
+ ### Failing
202
+
203
+ | You raise | Outcome |
204
+ |---|---|
205
+ | any exception | retried per the policy, then FAILED |
206
+ | `NonRetriableError` | FAILED immediately, remaining attempts skipped |
207
+ | `JobCancellationError` | CANCELLED, no retry, no `on_final_failure` |
208
+ | nothing | SUCCEEDED |
209
+
210
+ ## Configuration
211
+
212
+ ```python
213
+ SchedulerConfig(
214
+ dsn="postgresql://user:pass@host:5432/scheduler", # or dsn_from_jdbc(...)
215
+ node_id="billing-1",
216
+ default_queue="default",
217
+ default_max_attempts=3,
218
+ default_timeout_seconds=300,
219
+ default_retry_policy=ExponentialBackoff(max_attempts=3),
220
+ )
221
+
222
+ WorkerConfig(
223
+ node_id="billing-1",
224
+ node_tags=["eu-west"],
225
+ heartbeat_interval_seconds=30, # must be <= lock_duration / 3
226
+ lock_duration_seconds=90,
227
+ shutdown_timeout_seconds=30,
228
+ ).queue("billing", concurrency=8, prefetch=8)
229
+
230
+ RabbitConfig(url="amqp://scheduler:scheduler@localhost:5672/", queues=["billing"])
231
+ ```
232
+
233
+ Sharing `POSTGRES_URL` with the Kotlin services:
234
+
235
+ ```python
236
+ dsn = dsn_from_jdbc(os.environ["POSTGRES_URL"], os.environ["POSTGRES_USER"], os.environ["POSTGRES_PASSWORD"])
237
+ ```
238
+
239
+ ## Running Python and Kotlin side by side
240
+
241
+ Both clients read and write the same tables, and the dashboard shows their jobs together.
242
+ What does **not** cross the language boundary is the payload itself: a `payload_type` is a
243
+ Python class name here and a Kotlin FQN there, and a worker that picks up a type it does
244
+ not know marks the job FAILED rather than passing it on.
245
+
246
+ **So give each language its own queues.** Point Python handlers at `python`, `ml`, or
247
+ whichever names you like, and keep Kotlin workers on theirs:
248
+
249
+ ```python
250
+ RabbitConfig(url=AMQP_URL, queues=["ml"])
251
+ WorkerConfig(node_id="ml-1").queue("ml", concurrency=4)
252
+ ```
253
+
254
+ ```kotlin
255
+ schedulerRabbitModule { queues = listOf("default", "ml") } // infra must declare every queue
256
+ schedulerWorkerModule { queue("default", concurrency = 8) } // but only consumes its own
257
+ ```
258
+
259
+ The infra process needs every queue name in its `schedulerRabbitModule.queues` so the
260
+ topology exists; it does not need to consume them.
261
+
262
+ If you do want the two languages to run each other's jobs, pin `@job_type("<kotlin FQN>")`
263
+ and keep the JSON field names identical to the Kotlin data class — this SDK will encode and
264
+ decode it, but nothing checks that the two definitions still agree.
265
+
266
+ ## Guarantees
267
+
268
+ **At-least-once.** A job can run twice — a lease expiring during a long GC pause, a broker
269
+ redelivery, a network partition. `ctx.job_id` is stable across every attempt, so use it as
270
+ the idempotency key for anything with side effects:
271
+
272
+ ```python
273
+ await payments.charge(order_id, idempotency_key=str(ctx.job_id))
274
+ ```
275
+
276
+ **Leases, not locks.** A claimed job is held by `locked_until`, extended every
277
+ `heartbeat_interval_seconds`. If this process dies, the lease lapses and infra re-enqueues
278
+ the job — that is the recovery path, and it is why `heartbeat_interval` must stay at or
279
+ below a third of `lock_duration`.
280
+
281
+ **Schema evolution.** Adding a field with a default is safe. Removing one is safe — unknown
282
+ keys are ignored on decode. Renaming or retyping is not: version the payload
283
+ (`SendInvoiceV2`) and keep both handlers until the old jobs have drained. A payload that
284
+ cannot be decoded fails terminally without burning retries, since the stored bytes will
285
+ never change.
286
+
287
+ ## Development
288
+
289
+ ```bash
290
+ uv venv && uv pip install -e ".[dev]"
291
+ pytest tests/unit # no infrastructure needed
292
+ ruff check src tests examples scripts && mypy src
293
+ ```
294
+
295
+ Integration tests need a database with the migrations applied and a broker with the
296
+ delayed-message plugin. The whole suite runs in CI on every change under `clients/python`
297
+ (`.github/workflows/python-client.yml`); locally, bring the two up yourself. The commands
298
+ below assume a checkout of the repository, run from `clients/python`:
299
+
300
+ ```bash
301
+ docker run -d --name ts-pg -e POSTGRES_USER=scheduler -e POSTGRES_PASSWORD=scheduler \
302
+ -e POSTGRES_DB=scheduler -p 5432:5432 postgres:16-alpine
303
+
304
+ docker build -t taskscheduler-rabbit ../../docker/rabbitmq
305
+ docker run -d --name ts-rabbit -e RABBITMQ_DEFAULT_USER=scheduler \
306
+ -e RABBITMQ_DEFAULT_PASS=scheduler -p 5672:5672 taskscheduler-rabbit
307
+
308
+ python scripts/apply_migrations.py "postgresql://scheduler:scheduler@localhost:5432/scheduler"
309
+
310
+ TASKSCHEDULER_TEST_DSN="postgresql://scheduler:scheduler@localhost:5432/scheduler" \
311
+ TASKSCHEDULER_TEST_AMQP="amqp://scheduler:scheduler@localhost:5672/" \
312
+ pytest tests/integration
313
+ ```
314
+
315
+ `scripts/apply_migrations.py` replays the project's Flyway migrations without a JVM, so the
316
+ tests don't need a built `scheduler-infra` image. It is a development shortcut — in a real
317
+ deployment `scheduler-infra` owns the schema.
318
+
319
+ No Kotlin process runs during the tests: an `outbox_pump` fixture stands in for the infra
320
+ leader that would otherwise drain the outbox into RabbitMQ. Tests that expect a job to be
321
+ delivered more than once (retries, DAG promotions, paused-type redelivery) request it.
322
+
323
+ Alternatively `docker compose up -d` at the repo root brings up Postgres, RabbitMQ and a
324
+ real `scheduler-infra` — closer to production, but it needs the Gradle-built image
325
+ (`./gradlew :standalone-runner:dockerImage`).
@@ -0,0 +1,297 @@
1
+ # TaskScheduler — Python SDK
2
+
3
+ Async Python client for [TaskScheduler](https://github.com/4aK-Boris/TaskScheduler). A Python service becomes a
4
+ first-class participant in the same job system as your Kotlin services: it enqueues work,
5
+ runs handlers, and shows up on the same dashboard with the same retry, cancellation and
6
+ progress semantics.
7
+
8
+ ```python
9
+ from dataclasses import dataclass
10
+ from taskscheduler import Scheduler, SchedulerConfig, job_type
11
+
12
+ @job_type
13
+ @dataclass
14
+ class SendInvoice:
15
+ order_id: int
16
+
17
+ async with Scheduler(SchedulerConfig(dsn="postgresql://scheduler:scheduler@localhost/scheduler")) as s:
18
+ await s.enqueue(SendInvoice(order_id=42))
19
+ ```
20
+
21
+ ## How it fits together
22
+
23
+ The SDK speaks the same wire protocol as the Kotlin client — it is not a proxy in front of
24
+ it. PostgreSQL holds all job state; RabbitMQ carries nothing but a 16-byte job id as a
25
+ delivery hint.
26
+
27
+ ```
28
+ ┌──────────────────────────┐ ┌──────────────────────────┐
29
+ │ scheduler-infra (Kotlin) │ │ your Python service │
30
+ │ • owns the schema │ │ • Scheduler.enqueue() │
31
+ │ • outbox → RabbitMQ │ PG + │ • WorkerPool runs jobs │
32
+ │ • recurring cron │ Rabbit │ • heartbeats its lease │
33
+ │ • orphan recovery │◄──────►│ │
34
+ │ • dashboard :8080 │ │ │
35
+ └──────────────────────────┘ └──────────────────────────┘
36
+ └────────► PostgreSQL ◄────────┘
37
+ └────────► RabbitMQ ◄────────┘
38
+ ```
39
+
40
+ **`scheduler-infra` is required.** It owns the Flyway migrations and the background loops
41
+ that neither client implements: publishing the outbox to RabbitMQ, firing cron definitions,
42
+ recovering jobs whose worker died, and retention. This SDK verifies the schema version at
43
+ startup and refuses to run against a database that is too old.
44
+
45
+ ## Install
46
+
47
+ ```bash
48
+ pip install taskscheduler-client # or: uv pip install taskscheduler-client
49
+ ```
50
+
51
+ Requires Python 3.10+, and a RabbitMQ with the `rabbitmq_delayed_message_exchange` plugin
52
+ enabled (the same requirement the Kotlin side has — it is how delays and retry backoff work).
53
+
54
+ **Versioning.** The client shares one version with the rest of the project:
55
+ `taskscheduler-client 0.6.0` is the client for `scheduler-infra 0.6.0`, and the two are
56
+ released together. Run a client older than your infra and it may not know about a newer
57
+ column; newer, and it fails fast on the schema check. CI enforces that
58
+ `pyproject.toml`, `taskscheduler.__version__` and Gradle's `schedulerVersion` agree.
59
+
60
+ **On Windows**, select the other event loop before starting anything — psycopg cannot run on
61
+ the `ProactorEventLoop` that asyncio uses by default there:
62
+
63
+ ```python
64
+ if sys.platform == "win32":
65
+ asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
66
+ ```
67
+
68
+ The SDK raises a `ConfigurationError` naming this if you forget, rather than hanging on a
69
+ connection-pool timeout.
70
+
71
+ ## Producing jobs
72
+
73
+ A payload is a plain dataclass. `@job_type` gives it a stable name that is stored in
74
+ `job.payload_type`:
75
+
76
+ ```python
77
+ @job_type # -> "billing.jobs.SendInvoice"
78
+ @dataclass
79
+ class SendInvoice:
80
+ order_id: int
81
+ dry_run: bool = False
82
+
83
+ @job_type("billing.SendInvoice.v2") # pin it before renaming or moving the class
84
+ @dataclass
85
+ class SendInvoiceV2:
86
+ ...
87
+ ```
88
+
89
+ ```python
90
+ # now
91
+ await scheduler.enqueue(SendInvoice(order_id=42), queue="billing", priority=7)
92
+
93
+ # at a moment
94
+ await scheduler.schedule_at(SendReminder(order_id=42), at=datetime(2026, 6, 1, 10, tzinfo=timezone.utc))
95
+ await scheduler.schedule_in(SendReminder(order_id=42), delay=timedelta(days=1))
96
+
97
+ # at most one active job per key
98
+ await scheduler.enqueue_once(f"sync-user-{user_id}", SyncUser(user_id))
99
+
100
+ # strictly in order
101
+ await scheduler.chain(ExtractData(), TransformData(), LoadData())
102
+
103
+ # after a fan-out finishes
104
+ a = await scheduler.enqueue(LoadProductCache())
105
+ b = await scheduler.enqueue(LoadUserCache())
106
+ await scheduler.enqueue_after(StartPricingEngine(), wait_for=[a, b])
107
+
108
+ # on a cron
109
+ await scheduler.recurring("nightly-rollup", "0 3 * * *", NightlyRollup(), timezone_name="Europe/Berlin")
110
+ ```
111
+
112
+ Every enqueue writes the job row and its outbox row in **one transaction**, so a job
113
+ becomes visible only if your surrounding business transaction commits.
114
+
115
+ ## Consuming jobs
116
+
117
+ ```python
118
+ from taskscheduler import HandlerRegistry, JobContext, RabbitConfig, WorkerConfig, WorkerPool
119
+
120
+ registry = HandlerRegistry()
121
+
122
+ @registry.handler(SendInvoice, retry_policy=ExponentialBackoff(max_attempts=5))
123
+ async def send_invoice(ctx: JobContext, job: SendInvoice) -> None:
124
+ await billing.send(job.order_id, idempotency_key=str(ctx.job_id))
125
+
126
+ worker = WorkerPool(
127
+ scheduler_config=SchedulerConfig(dsn=DSN, node_id="billing-1"),
128
+ worker_config=WorkerConfig(node_id="billing-1").queue("billing", concurrency=8),
129
+ rabbit_config=RabbitConfig(url=AMQP_URL, queues=["billing"]),
130
+ registry=registry,
131
+ )
132
+
133
+ async with worker:
134
+ await asyncio.Event().wait() # run until the process is stopped
135
+ ```
136
+
137
+ Class-based handlers work too, when you need `on_final_failure` or dependency injection:
138
+
139
+ ```python
140
+ class SendInvoiceHandler(JobHandler[SendInvoice]):
141
+ payload = SendInvoice
142
+ retry_policy = ExponentialBackoff(max_attempts=5)
143
+
144
+ def __init__(self, billing: BillingClient) -> None:
145
+ self._billing = billing
146
+
147
+ async def execute(self, ctx: JobContext, job: SendInvoice) -> None:
148
+ await self._billing.send(job.order_id)
149
+
150
+ async def on_final_failure(self, ctx: JobContext, job: SendInvoice, error: BaseException) -> None:
151
+ await alerts.page(f"invoice {job.order_id} failed permanently")
152
+
153
+ registry.register(SendInvoiceHandler(billing))
154
+ ```
155
+
156
+ ### Progress and cancellation
157
+
158
+ ```python
159
+ @registry.handler(ReindexCatalog)
160
+ async def reindex(ctx: JobContext, job: ReindexCatalog) -> None:
161
+ bar = ctx.progress_bar(len(job.product_ids))
162
+ for product_id in job.product_ids:
163
+ if await ctx.is_cancellation_requested():
164
+ raise JobCancellationError() # ends as CANCELLED, not FAILED
165
+ await index(product_id)
166
+ await bar.succeeded()
167
+ ```
168
+
169
+ Progress writes are throttled to one per second, so calling them in a tight loop is fine.
170
+ A cancelled job that ignores the flag is cancelled outright after
171
+ `WorkerConfig.cancel_grace_seconds`.
172
+
173
+ ### Failing
174
+
175
+ | You raise | Outcome |
176
+ |---|---|
177
+ | any exception | retried per the policy, then FAILED |
178
+ | `NonRetriableError` | FAILED immediately, remaining attempts skipped |
179
+ | `JobCancellationError` | CANCELLED, no retry, no `on_final_failure` |
180
+ | nothing | SUCCEEDED |
181
+
182
+ ## Configuration
183
+
184
+ ```python
185
+ SchedulerConfig(
186
+ dsn="postgresql://user:pass@host:5432/scheduler", # or dsn_from_jdbc(...)
187
+ node_id="billing-1",
188
+ default_queue="default",
189
+ default_max_attempts=3,
190
+ default_timeout_seconds=300,
191
+ default_retry_policy=ExponentialBackoff(max_attempts=3),
192
+ )
193
+
194
+ WorkerConfig(
195
+ node_id="billing-1",
196
+ node_tags=["eu-west"],
197
+ heartbeat_interval_seconds=30, # must be <= lock_duration / 3
198
+ lock_duration_seconds=90,
199
+ shutdown_timeout_seconds=30,
200
+ ).queue("billing", concurrency=8, prefetch=8)
201
+
202
+ RabbitConfig(url="amqp://scheduler:scheduler@localhost:5672/", queues=["billing"])
203
+ ```
204
+
205
+ Sharing `POSTGRES_URL` with the Kotlin services:
206
+
207
+ ```python
208
+ dsn = dsn_from_jdbc(os.environ["POSTGRES_URL"], os.environ["POSTGRES_USER"], os.environ["POSTGRES_PASSWORD"])
209
+ ```
210
+
211
+ ## Running Python and Kotlin side by side
212
+
213
+ Both clients read and write the same tables, and the dashboard shows their jobs together.
214
+ What does **not** cross the language boundary is the payload itself: a `payload_type` is a
215
+ Python class name here and a Kotlin FQN there, and a worker that picks up a type it does
216
+ not know marks the job FAILED rather than passing it on.
217
+
218
+ **So give each language its own queues.** Point Python handlers at `python`, `ml`, or
219
+ whichever names you like, and keep Kotlin workers on theirs:
220
+
221
+ ```python
222
+ RabbitConfig(url=AMQP_URL, queues=["ml"])
223
+ WorkerConfig(node_id="ml-1").queue("ml", concurrency=4)
224
+ ```
225
+
226
+ ```kotlin
227
+ schedulerRabbitModule { queues = listOf("default", "ml") } // infra must declare every queue
228
+ schedulerWorkerModule { queue("default", concurrency = 8) } // but only consumes its own
229
+ ```
230
+
231
+ The infra process needs every queue name in its `schedulerRabbitModule.queues` so the
232
+ topology exists; it does not need to consume them.
233
+
234
+ If you do want the two languages to run each other's jobs, pin `@job_type("<kotlin FQN>")`
235
+ and keep the JSON field names identical to the Kotlin data class — this SDK will encode and
236
+ decode it, but nothing checks that the two definitions still agree.
237
+
238
+ ## Guarantees
239
+
240
+ **At-least-once.** A job can run twice — a lease expiring during a long GC pause, a broker
241
+ redelivery, a network partition. `ctx.job_id` is stable across every attempt, so use it as
242
+ the idempotency key for anything with side effects:
243
+
244
+ ```python
245
+ await payments.charge(order_id, idempotency_key=str(ctx.job_id))
246
+ ```
247
+
248
+ **Leases, not locks.** A claimed job is held by `locked_until`, extended every
249
+ `heartbeat_interval_seconds`. If this process dies, the lease lapses and infra re-enqueues
250
+ the job — that is the recovery path, and it is why `heartbeat_interval` must stay at or
251
+ below a third of `lock_duration`.
252
+
253
+ **Schema evolution.** Adding a field with a default is safe. Removing one is safe — unknown
254
+ keys are ignored on decode. Renaming or retyping is not: version the payload
255
+ (`SendInvoiceV2`) and keep both handlers until the old jobs have drained. A payload that
256
+ cannot be decoded fails terminally without burning retries, since the stored bytes will
257
+ never change.
258
+
259
+ ## Development
260
+
261
+ ```bash
262
+ uv venv && uv pip install -e ".[dev]"
263
+ pytest tests/unit # no infrastructure needed
264
+ ruff check src tests examples scripts && mypy src
265
+ ```
266
+
267
+ Integration tests need a database with the migrations applied and a broker with the
268
+ delayed-message plugin. The whole suite runs in CI on every change under `clients/python`
269
+ (`.github/workflows/python-client.yml`); locally, bring the two up yourself. The commands
270
+ below assume a checkout of the repository, run from `clients/python`:
271
+
272
+ ```bash
273
+ docker run -d --name ts-pg -e POSTGRES_USER=scheduler -e POSTGRES_PASSWORD=scheduler \
274
+ -e POSTGRES_DB=scheduler -p 5432:5432 postgres:16-alpine
275
+
276
+ docker build -t taskscheduler-rabbit ../../docker/rabbitmq
277
+ docker run -d --name ts-rabbit -e RABBITMQ_DEFAULT_USER=scheduler \
278
+ -e RABBITMQ_DEFAULT_PASS=scheduler -p 5672:5672 taskscheduler-rabbit
279
+
280
+ python scripts/apply_migrations.py "postgresql://scheduler:scheduler@localhost:5432/scheduler"
281
+
282
+ TASKSCHEDULER_TEST_DSN="postgresql://scheduler:scheduler@localhost:5432/scheduler" \
283
+ TASKSCHEDULER_TEST_AMQP="amqp://scheduler:scheduler@localhost:5672/" \
284
+ pytest tests/integration
285
+ ```
286
+
287
+ `scripts/apply_migrations.py` replays the project's Flyway migrations without a JVM, so the
288
+ tests don't need a built `scheduler-infra` image. It is a development shortcut — in a real
289
+ deployment `scheduler-infra` owns the schema.
290
+
291
+ No Kotlin process runs during the tests: an `outbox_pump` fixture stands in for the infra
292
+ leader that would otherwise drain the outbox into RabbitMQ. Tests that expect a job to be
293
+ delivered more than once (retries, DAG promotions, paused-type redelivery) request it.
294
+
295
+ Alternatively `docker compose up -d` at the repo root brings up Postgres, RabbitMQ and a
296
+ real `scheduler-infra` — closer to production, but it needs the Gradle-built image
297
+ (`./gradlew :standalone-runner:dockerImage`).