django-gobeat 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 BAPP
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,374 @@
1
+ Metadata-Version: 2.4
2
+ Name: django-gobeat
3
+ Version: 0.1.0
4
+ Summary: cron.py periodic jobs for Django + the companion app of gobeat, a small Go replacement for the celery beat process
5
+ License-Expression: MIT
6
+ Project-URL: Homepage, https://bapp-open.github.io/django-celery-beat-go/
7
+ Project-URL: Repository, https://github.com/bapp-open/django-celery-beat-go
8
+ Project-URL: Issues, https://github.com/bapp-open/django-celery-beat-go/issues
9
+ Keywords: django,celery,beat,cron,scheduler,periodic tasks,django-celery-beat
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Framework :: Django
12
+ Classifier: Framework :: Django :: 5.2
13
+ Classifier: Framework :: Django :: 6.0
14
+ Classifier: Framework :: Celery
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Operating System :: OS Independent
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Topic :: System :: Distributed Computing
19
+ Requires-Python: >=3.10
20
+ Description-Content-Type: text/markdown
21
+ License-File: LICENSE
22
+ Requires-Dist: Django>=5.2
23
+ Requires-Dist: celery<5.8,>=5.7
24
+ Provides-Extra: django-celery-beat
25
+ Requires-Dist: django-celery-beat<2.10,>=2.8; extra == "django-celery-beat"
26
+ Dynamic: license-file
27
+
28
+ # django-celery-beat-go (gobeat)
29
+
30
+ [![ci](https://github.com/bapp-open/django-celery-beat-go/actions/workflows/ci.yml/badge.svg)](https://github.com/bapp-open/django-celery-beat-go/actions/workflows/ci.yml)
31
+ [![PyPI](https://img.shields.io/pypi/v/django-gobeat)](https://pypi.org/project/django-gobeat/)
32
+ [![image](https://img.shields.io/badge/ghcr.io-django--celery--beat--go-blue)](https://github.com/bapp-open/django-celery-beat-go/pkgs/container/django-celery-beat-go)
33
+
34
+ **gobeat** is a small Go service that replaces the `celery beat` process
35
+ for Django projects. Workers and tasks stay exactly as they are; only the
36
+ scheduler process changes. It comes with **django-gobeat**, a Django app
37
+ that lets you declare periodic jobs in each app's `cron.py`.
38
+
39
+ ```sh
40
+ pip install django-gobeat # the Django app
41
+ docker run -e DATABASE_URL=postgres://… ghcr.io/bapp-open/django-celery-beat-go # the scheduler
42
+ ```
43
+
44
+ ## What it does
45
+
46
+ * **Schedules periodic Celery tasks** from two sources:
47
+ `cron.py` classes extending `CronInterface` (auto-discovered like
48
+ `models.py`, stored in gobeat's own table, no django-celery-beat
49
+ needed), and — optionally — django-celery-beat's `PeriodicTask` rows
50
+ and admin, with behaviour identical to its `DatabaseScheduler`.
51
+ * **Publishes the same messages Celery does** (protocol v2) to **Redis**
52
+ or **RabbitMQ**: byte-identical to kombu's, verified against real
53
+ Celery.
54
+ * **Never double-fires or silently loses a run**: every run is claimed in
55
+ Postgres before it is published, kept in a journal until the broker
56
+ confirms it, and only one replica is active at a time.
57
+ * **Stays small**: one static binary on an empty image (16 MB), about
58
+ 12 MB of memory.
59
+ * **Shadow mode**: runs beside Python beat without writing anything, and
60
+ a tool compares the two dispatch logs before you switch.
61
+
62
+ ## What it does not do
63
+
64
+ * **Only PostgreSQL** as the Django database; **only Redis and RabbitMQ**
65
+ as brokers (no SQS, no Redis Sentinel/Cluster, one broker URL).
66
+ * **Only the latest stable Celery** (5.7.x). It refuses to start on other
67
+ versions instead of guessing.
68
+ * **JSON task messages only**; no pickle/yaml/msgpack serializers, no
69
+ compression other than gzip/zlib.
70
+ * **No solar schedules.**
71
+ * **It is not a worker and does not run your code**: it never imports the
72
+ project, so publish signal handlers (Sentry/OpenTelemetry header
73
+ injection), `task-sent` events, custom `Task.apply_async` and router
74
+ functions that depend on call arguments are not reproduced.
75
+ * **No built-in result handling**; it only sends tasks.
76
+
77
+ Anything unsupported is reported with an error that names the task — it
78
+ is skipped, never approximated. Details:
79
+ [Known differences](#known-differences-from-python-beat).
80
+
81
+ Reference versions: **Celery 5.7.0, kombu 5.7.0, django-celery-beat 2.9.0**
82
+ — see [Version support](#version-support). Design notes and the source
83
+ map (which Python file each behaviour was copied from) are in
84
+ [docs/DESIGN.md](docs/DESIGN.md).
85
+
86
+ ## How it fits together
87
+
88
+ ```
89
+ manage.py migrate (Django project, at deploy; syncs on post_migrate)
90
+ │ writes one JSON row: broker, timezone, queues, per-task routing
91
+ ▼
92
+ Postgres ◀── LISTEN/NOTIFY + polling ── gobeat container ──▶ Redis / RabbitMQ ──▶ workers
93
+ (django_celery_beat tables, settings row, dispatch journal)
94
+ ```
95
+
96
+ gobeat never imports your code. Everything that needs the Celery app
97
+ (`task_routes`, decorator options, queue declarations, `beat_schedule`)
98
+ is resolved by Celery itself inside `gobeat_sync`.
99
+
100
+ ## Periodic jobs in `cron.py`
101
+
102
+ ```python
103
+ # myapp/cron.py - discovered automatically for every app in INSTALLED_APPS
104
+ from datetime import timedelta
105
+ from django_gobeat import CronInterface, crontab
106
+
107
+ class NightlyCleanup(CronInterface):
108
+ schedule = crontab(minute=0, hour=4) # celery.schedules.crontab, a timedelta, or seconds
109
+ timezone = "Europe/Bucharest" # crontab zone; default: CELERY_TIMEZONE
110
+ queue = "maintenance" # also: exchange, routing_key, priority, expires, headers
111
+
112
+ def run(self): # executed by a Celery worker
113
+ ...
114
+
115
+ class Heartbeat(CronInterface):
116
+ schedule = timedelta(seconds=30)
117
+ task = "myapp.tasks.ping" # schedule an existing task instead of run()
118
+ kwargs = {"source": "cron"}
119
+ ```
120
+
121
+ * A class with `run()` is registered as a Celery task named after the job
122
+ (`myapp.cron.NightlyCleanup`, or `name = "..."`); workers find it
123
+ because discovery happens in `AppConfig.ready()`.
124
+ * Project conventions go into your own base class: mark it
125
+ `abstract = True` and override `get_schedule()`, `get_args()`,
126
+ `get_kwargs()`, `get_headers()`, `is_enabled()` or `run()`.
127
+ * After `migrate` (or with `manage.py gobeat_sync`) the discovered jobs
128
+ are written to `django_gobeat_job` (creating, updating and removing rows to match the
129
+ code; run counts and the admin's *paused* switch are kept). Jobs can
130
+ also be created by hand in the admin (`source = admin`).
131
+ * Without django-celery-beat, entries of `app.conf.beat_schedule` are
132
+ written to the same table.
133
+ * Semantics are Celery's: crontabs are evaluated exactly like
134
+ `celery.schedules.crontab` in the job's zone; an interval job first
135
+ runs one interval after it was defined and then every interval.
136
+
137
+ ## Migration steps
138
+
139
+ 1. `pip install django-gobeat` and add the app (`django-celery-beat` may
140
+ stay installed or not — gobeat uses its tables only when it is):
141
+
142
+ ```python
143
+ INSTALLED_APPS += ["django_gobeat"]
144
+ # optional: GOBEAT_CELERY_APP = "proj.celery:app" (default: celery.current_app)
145
+ ```
146
+
147
+ 2. Run `migrate` on every deploy, as you already do:
148
+
149
+ ```sh
150
+ python manage.py migrate
151
+ ```
152
+
153
+ django_gobeat hooks `post_migrate` (which fires even when there is
154
+ nothing to migrate) and then writes the settings row, the `cron.py`
155
+ jobs and — as Python beat does on start-up — code-defined
156
+ `beat_schedule` entries. Warnings are printed for anything it cannot
157
+ resolve exactly; a failure is reported but never breaks the migration.
158
+ The same step is available as `python manage.py gobeat_sync`
159
+ (`--dry-run` prints the settings document, `--strict` fails on
160
+ warnings). Settings: `GOBEAT_AUTO_SYNC = False` disables the hook,
161
+ `GOBEAT_STORE_BROKER_URL = False` keeps the broker URL out of the row,
162
+ `GOBEAT_CELERY_APP = "proj.celery:app"` names the app explicitly.
163
+
164
+ 3. **Shadow first.** Keep Python beat running and start gobeat beside it
165
+ in dry-run mode; it logs what it would dispatch and writes nothing:
166
+
167
+ ```sh
168
+ docker run -e DATABASE_URL=postgres://… -e GOBEAT_DRY_RUN=1 ghcr.io/bapp-open/django-celery-beat-go
169
+ ```
170
+
171
+ After a day, compare the two logs:
172
+
173
+ ```sh
174
+ gobeat shadow-diff --python-log beat.log --gobeat-log gobeat.log \
175
+ --python-tz Europe/Bucharest --tolerance 2s
176
+ ```
177
+
178
+ (`beat.log` is `celery beat --loglevel=INFO` output in Celery's default
179
+ format; `--python-tz` is the zone of its timestamps, normally Django's
180
+ `TIME_ZONE`.) Exit status 0 means every dispatch has a partner.
181
+
182
+ 4. Cut over: stop Python beat, then start gobeat without `GOBEAT_DRY_RUN`.
183
+ Do not run both live — Python beat does not take part in the claim.
184
+
185
+ ## Rollback
186
+
187
+ Stop gobeat, start `celery beat` again. gobeat keeps `last_run_at` and
188
+ `total_run_count` in the django-celery-beat tables up to date, so Python
189
+ beat continues from where gobeat stopped. Jobs that exist only in
190
+ `cron.py` / `django_gobeat_job` are not known to Python beat: list them in
191
+ `beat_schedule` if you need them during a rollback. The `django_gobeat_*`
192
+ tables can stay or be dropped with `manage.py migrate django_gobeat zero`.
193
+
194
+ ## Configuration
195
+
196
+ One container per project. Only `DATABASE_URL` is required; everything
197
+ else comes from the settings row and environment variables override it.
198
+
199
+ | Variable | Default | Meaning |
200
+ |---|---|---|
201
+ | `DATABASE_URL` | – (required) | Postgres URL of the Django database |
202
+ | `GOBEAT_DRY_RUN` | `0` | log dispatches, write and publish nothing (shadow mode) |
203
+ | `GOBEAT_MISSED_RUN_POLICY` | `catch_up_once` | after downtime: one run for the gap (what Python beat does), or `skip` |
204
+ | `GOBEAT_MISSED_RUN_GRACE` | `60s` | lateness beyond which `skip` skips a run |
205
+ | `GOBEAT_PUBLISH_TIMEOUT` | `10s` | per publish attempt |
206
+ | `GOBEAT_PUBLISH_MAX_AGE` | `1h` | keep retrying a claimed run this long, then abandon it loudly |
207
+ | `GOBEAT_JOURNAL_RETENTION` | `24h` | keep published journal rows this long |
208
+ | `GOBEAT_HTTP_ADDR` | `:9099` | `/healthz` and `/metrics` (`off` disables) |
209
+ | `GOBEAT_HEALTH_MAX_AGE` | `30s` | `/healthz` fails when the loop has not ticked for this long |
210
+ | `GOBEAT_STANDBY_INTERVAL` | `5s` | how often a standby replica retries the lock |
211
+ | `GOBEAT_LOCK_KEY` | `0x676f62656174` | advisory lock key |
212
+ | `GOBEAT_SETTINGS_KEY` | `default` | settings row to use |
213
+ | `GOBEAT_LOG_LEVEL` / `GOBEAT_LOG_FORMAT` | `info` / `json` | |
214
+ | `GOBEAT_ALLOW_UNSUPPORTED_VERSIONS` | `0` | run against untested Celery/django-celery-beat versions |
215
+ | **overrides of the settings row** | | |
216
+ | `GOBEAT_BROKER_URL` | row | `redis://`, `rediss://`, `amqp://`, `amqps://` |
217
+ | `GOBEAT_BROKER_TRANSPORT_OPTIONS` | row | JSON (`priority_steps`, `sep`, `global_keyprefix`) |
218
+ | `GOBEAT_TIMEZONE` | row | `CELERY_TIMEZONE` |
219
+ | `GOBEAT_MAX_LOOP_INTERVAL` | row (5) | seconds, `beat_max_loop_interval` |
220
+ | `GOBEAT_CRON_STARTING_DEADLINE` | row | seconds, `beat_cron_starting_deadline` |
221
+ | `GOBEAT_SETTINGS_OVERRIDE` | – | JSON object merged over the settings document |
222
+
223
+ The broker URL (with its password) is part of the settings row; use
224
+ `GOBEAT_STORE_BROKER_URL = False` (or `gobeat_sync --no-broker-url`) and
225
+ `GOBEAT_BROKER_URL` to keep it out of the database.
226
+
227
+ gobeat **refuses to start**, with an explanatory error, when the settings
228
+ row is missing, the django-celery-beat schema is unknown, the versions are
229
+ unsupported or the configuration cannot be reproduced (see below). Once
230
+ started it never exits on database or broker failures; it retries with
231
+ backoff.
232
+
233
+ ### Endpoints
234
+
235
+ * `GET /healthz` – 200 while the loop ticks (leader, standby or retrying
236
+ a dead database), 503 when it has not ticked for `GOBEAT_HEALTH_MAX_AGE`.
237
+ The image's `HEALTHCHECK` uses `gobeat healthcheck`.
238
+ * `GET /metrics` – Prometheus text: `gobeat_dispatch_total`,
239
+ `gobeat_dispatch_lateness_seconds` (histogram), `gobeat_errors_total{kind}`,
240
+ `gobeat_reloads_total{reason}`, `gobeat_claim_conflicts_total`,
241
+ `gobeat_skipped_runs_total`, `gobeat_abandoned_total`,
242
+ `gobeat_publish_retries_total`, `gobeat_pending_publishes`,
243
+ `gobeat_leader`, `gobeat_tasks`, `gobeat_tasks_refused`,
244
+ `gobeat_last_tick_timestamp_seconds`.
245
+
246
+ ## Delivery guarantees
247
+
248
+ * **Claim before publish.** One SQL statement updates `last_run_at` /
249
+ `total_run_count` only if `last_run_at` is still what gobeat saw, and
250
+ inserts the rendered message into `django_gobeat_dispatch`. Only then is
251
+ the message published; the row is marked when the broker accepted it
252
+ (Redis reply / RabbitMQ publisher confirm).
253
+ * **No lost claimed runs.** Broker down: runs keep being claimed and wait
254
+ in the journal, published in order when the broker is back. Crash
255
+ between claim and publish: the next leader publishes the orphan first.
256
+ A run is given up only after `GOBEAT_PUBLISH_MAX_AGE`, with an error log
257
+ and `gobeat_abandoned_total`.
258
+ * **At-least-once, one edge.** If the process dies after the broker
259
+ accepted a message but before the journal row was marked, that one run
260
+ is published again after restart — with the same task id.
261
+ * **Deterministic task ids**: UUIDv5 of (periodic task id, scheduled
262
+ instant), so a duplicate is recognisable; the journal's unique task id
263
+ also stops the same slot from being claimed twice.
264
+ * **One leader.** A second replica stands by on the advisory lock and
265
+ takes over within `GOBEAT_STANDBY_INTERVAL` of the leader's connection
266
+ ending.
267
+ * **After downtime** (gobeat, database, or a forward clock jump): one
268
+ catch-up run per task, then the normal cadence (`catch_up_once`), or no
269
+ catch-up (`skip`). A clock stepping backwards delays runs; it never
270
+ repeats one.
271
+
272
+ ## Known differences from Python beat
273
+
274
+ Deliberate:
275
+
276
+ 1. **State is in the database.** Python writes `last_run_at` every three
277
+ minutes; gobeat writes it with every claim.
278
+ 2. **One-off tasks are disabled right after their run**, not at the next
279
+ evaluation; the resulting row is the same (`enabled=False`,
280
+ `total_run_count=0`).
281
+ 3. **No crontab hour window on load.** django-celery-beat 2.8 only loads
282
+ numeric-hour crontabs within ±2 h of the current hour. A side effect:
283
+ a run missed during downtime is caught up when its hour window opens
284
+ the next day. gobeat catches up at start-up.
285
+ 4. **Publish failures are retried** instead of being logged and dropped,
286
+ and messages are published with publisher confirms and `mandatory`
287
+ (RabbitMQ) — an unroutable message is an error, not a silent drop.
288
+ 5. **Refused, never guessed** (an error names the task; it is not
289
+ scheduled): solar schedules; non-JSON serializers; compression other
290
+ than gzip/zlib; `args`/`kwargs`/`headers` that are not a JSON
291
+ array/object or contain kombu `__type__` markers; non-positive
292
+ intervals; crontabs that never match (February 31st, a `ValueError`
293
+ in Celery too); routing that ends on an empty queue name (Python
294
+ publishes to a queue called `""`); fan-out exchanges on Redis.
295
+ Python beat *disables* a task whose JSON does not parse; gobeat only
296
+ skips it.
297
+ 6. **Refused at start-up**: `USE_TZ` ≠ `DJANGO_CELERY_BEAT_TZ_AWARE`,
298
+ `enable_utc=False` without `CELERY_TIMEZONE`, `task_always_eager`,
299
+ task protocol 1, several broker URLs, other transports.
300
+
301
+ Not reproduced (things only a Python process can do):
302
+
303
+ * `before_task_publish` / `after_task_publish` signal handlers
304
+ (Sentry/OpenTelemetry header injection), `task-sent` events, custom
305
+ `Task.apply_async` / `shadow_name`, argument checking (`typing=True`).
306
+ * Router *functions* in `task_routes` are evaluated once per task by
307
+ `gobeat_sync` with empty arguments (a warning is printed); routes that
308
+ depend on call arguments are not reproduced.
309
+ * `reply_to` is a random id; no reply queue is declared for the RPC
310
+ result backend.
311
+ * gzip/zlib bodies decompress to the identical bytes but the compressed
312
+ stream itself is produced by Go's zlib.
313
+ * AMQP: content header and body are byte-identical; the `Basic.Publish`
314
+ frame additionally sets `mandatory`.
315
+
316
+ Inherited from Celery on purpose (the reference version does this):
317
+
318
+ * Crontabs are wall-clock in their own time zone: a time that does not
319
+ exist on the spring-forward day runs an hour later, a repeated time
320
+ runs once, and an every-minute crontab in a DST zone skips the repeated
321
+ hour in autumn. Intervals are elapsed time and are not affected by DST.
322
+ * A crontab with day-of-month *and* day-of-week needs both to match.
323
+ * A never-run task counts from `date_changed`; with `start_time` it first
324
+ runs at `start_time`.
325
+
326
+ ## Version support
327
+
328
+ gobeat checks the versions recorded by `gobeat_sync` and the applied
329
+ django-celery-beat migrations before doing anything.
330
+
331
+ | | status |
332
+ |---|---|
333
+ | Celery 5.7.x | supported (reference: 5.7.0) |
334
+ | django-celery-beat 2.8.x, 2.9.x | supported: goldens regenerated with 2.8.1 and 2.9.0 are byte-identical |
335
+ | Celery ≤ 5.6 | refused: crontab arithmetic around DST, interval arithmetic, `saferepr`, the message headers and beat's tick differ from 5.7 |
336
+ | django-celery-beat ≤ 2.7 | refused: a never-run task starts from "now" instead of `date_changed` |
337
+
338
+ When a new Celery release appears: bump `tests/requirements.txt`, run
339
+ `make golden soak-golden` and `make ci`; every behavioural change shows up
340
+ as a failing golden case.
341
+
342
+ ## Testing
343
+
344
+ `make ci` runs everything; the pieces:
345
+
346
+ | | command | what |
347
+ |---|---|---|
348
+ | unit | `make test` | schedule maths, crontab parsing, time zones, message encoding, config, HTTP, shadow tool — with `-race` and a goroutine-leak check in every package |
349
+ | differential | `make test` (replay), `make golden` (regenerate) | 70,000 schedule cases, 6,000 JSON/repr cases and 6,000 messages produced by real Celery/kombu/django-celery-beat (`tests/differential`); messages are captured from real Redis and RabbitMQ publishes |
350
+ | fuzz | `make fuzz` | crontab parser, crontab evaluation, JSON loader, message encoder |
351
+ | soak | `make test`, `make soak-golden` | real `DatabaseScheduler.tick()` on a fake clock vs gobeat's simulation over 30 simulated days: identical dispatch counts for every task (1,000 tasks, 1.24 M dispatches; and 10,000 tasks, 12.4 M dispatches, with a flat heap) |
352
+ | version matrix | `make matrix` | goldens regenerated under each supported version must be byte-identical |
353
+ | Django app | `make pytest` | `cron.py` discovery, `CronInterface`, job sync |
354
+ | integration | `make integration` | docker compose: Postgres, Redis, RabbitMQ, a Django project, real workers — on both brokers, plus a pass with django-celery-beat not installed |
355
+ | failure injection | `make integration` | database down/up, broker down/up, SIGKILL between claim and publish and between publish and journal mark, network partition, two instances with failover, missed-run policies; clock jumps on a fake clock in `internal/scheduler` |
356
+ | shadow | `make integration` | Python beat and gobeat dry-run side by side, logs compared with `gobeat shadow-diff` |
357
+
358
+ ## Releasing
359
+
360
+ 1. Set the new version in `pyproject.toml` and `django_gobeat/__init__.py`.
361
+ 2. Tag and push: `git tag v0.2.0 && git push origin v0.2.0`.
362
+
363
+ The `release` workflow then runs the test-suite, pushes the image to
364
+ `ghcr.io/bapp-open/django-celery-beat-go` (`:0.2.0`, `:0.2`, `:latest`,
365
+ amd64 + arm64), publishes `django-gobeat` to PyPI (trusted publishing) and
366
+ attaches static binaries to the GitHub release.
367
+
368
+ ## Development
369
+
370
+ ```sh
371
+ make build # bin/gobeat (static)
372
+ make test # needs only Go; golden files are committed
373
+ make docker # the image
374
+ ```