django-overseer 0.1.0__tar.gz

django_overseer-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[co]
+*.egg-info/
+/dist/
+/build/
+/.coverage*
+/htmlcov/
+/.pytest_cache/
+/.ruff_cache/
+/site/
+.DS_Store
+/docs/_build/

django_overseer-0.1.0/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+## [0.1.0] - 2026-10-07
+
+First release.
+
+- Recording of every `django.tasks` enqueue, start and finish into `Job`, `Run` and
+  `Worker` rows through the framework's signals; works with any backend.
+- `overseer.task`: retries with exponential, linear or constant backoff, jitter, cap,
+  `retry_on` filters, per-task timeouts, tags, and `unique` enqueues enforced by a database
+  constraint.
+- `overseer.schedule`: cron expressions (dependency-free parser, timezone aware) and fixed
+  intervals, synced into the `Schedule` table and safe across several scheduler processes.
+- `overseer_scheduler`, `overseer_worker` (heartbeats), `overseer_rescue`,
+  `overseer_rollup`, `overseer_alerts`, `overseer_prune` and `overseer_sync_schedules`
+  management commands.
+- Rescue of abandoned runs with backend reset for `django-tasks-db`.
+- Dashboard: overview, queues, tasks, jobs, job detail, failed, schedules, workers, metrics
+  and alerts pages with retry / cancel / dismiss / pause / run-now actions, auto refresh and
+  a JSON API with a health endpoint.
+- Alerts for failure rate, queue wait, queue depth and offline workers with cooldowns,
+  resolution, e-mail, Slack and custom notifiers, plus `alert_fired`, `alert_resolved` and
+  `job_failed` signals.
+- Per-minute metric rollups and retention pruning.
+- System checks for settings and non-deferring backends.
+- Test suite in four layers (unit, real-process integration, HTTP end-to-end, performance
+  budgets) on SQLite and PostgreSQL, Django 6.0 and 6.1.

django_overseer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 mohamed ali
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

django_overseer-0.1.0/PKG-INFO

@@ -0,0 +1,351 @@
+Metadata-Version: 2.5
+Name: django-overseer
+Version: 0.1.0
+Summary: Dashboard, retries and schedules for Django's built-in Tasks framework
+Project-URL: Homepage, https://github.com/mohamed-alired/django-overseer
+Project-URL: Repository, https://github.com/mohamed-alired/django-overseer.git
+Project-URL: Issues, https://github.com/mohamed-alired/django-overseer/issues
+Project-URL: Changelog, https://github.com/mohamed-alired/django-overseer/blob/main/CHANGELOG.md
+Author-email: mohamed ali <mohamed.ali@redwoodcompliance.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: background,cron,dashboard,django,queue,retries,scheduler,tasks
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Web Environment
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 6.0
+Classifier: Framework :: Django :: 6.1
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.12
+Requires-Dist: django>=6.0
+Provides-Extra: db
+Requires-Dist: django-tasks-db>=0.13; extra == 'db'
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: django-tasks-db>=0.13; extra == 'dev'
+Requires-Dist: freezegun>=1.5; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest-django>=4.9; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: requests>=2.31; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: test
+Requires-Dist: django-tasks-db>=0.13; extra == 'test'
+Requires-Dist: freezegun>=1.5; extra == 'test'
+Requires-Dist: pytest-cov; extra == 'test'
+Requires-Dist: pytest-django>=4.9; extra == 'test'
+Requires-Dist: pytest>=8; extra == 'test'
+Requires-Dist: requests>=2.31; extra == 'test'
+Description-Content-Type: text/markdown
+
+# django-overseer
+
+**Dashboard, retries, schedules and alerts for Django's built-in Tasks framework.**
+
+Django 6 ships a Tasks framework (`django.tasks`) and a database-backed worker
+(`django-tasks-db`), but no way to *see* what your tasks are doing, no retries, no cron and
+no alarm when a queue backs up. Overseer adds the operational layer that every other
+ecosystem takes for granted (Sidekiq's web UI, Celery's Flower, Oban's dashboard) without
+adding a broker, a JavaScript build or a second framework.
+
+- **Dashboard**: overview, queues, tasks, jobs with filters and search, job detail with the
+  full attempt chain and tracebacks, failed jobs with retry / dismiss, schedules, workers,
+  per-minute metrics and alerts. Plain Django views and templates, auto-refreshing, no
+  external assets.
+- **Retries**: `@overseer.task(retries=3, backoff="exponential")` with constant, linear or
+  exponential backoff, jitter, a cap, `retry_on=` exception filters and per-task timeouts.
+  Retries are real re-enqueues through `django.tasks`, deferred with `run_after` on backends
+  that support it.
+- **Unique tasks**: `unique=True` (or a key callable) collapses identical pending enqueues,
+  enforced by a database constraint so concurrent enqueues cannot both win.
+- **Schedules**: `@overseer.schedule("*/5 * * * *")` or `every=300`, synced into the database,
+  pausable and triggerable from the dashboard, with a scheduler that is safe to run on
+  several hosts at once (`SELECT ... FOR UPDATE SKIP LOCKED`).
+- **Rescue**: runs whose worker died are detected by timeout, marked abandoned, reset in the
+  backend and retried under the task's policy.
+- **Workers**: `overseer_worker` wraps `db_worker` with heartbeats, so the dashboard shows
+  which workers are alive, on which host, processing what.
+- **Alerts**: failure rate, queue wait, queue depth and offline workers, with cooldowns,
+  automatic resolution, e-mail, Slack and custom notifiers, plus Django signals.
+- **JSON API** and a `/api/health/` endpoint for your own monitoring.
+- **Retention**: per-minute metric rollups and pruning of old jobs, runs and alerts.
+
+Everything is recorded through the three `django.tasks` signals, so Overseer works with
+**any** task backend. Cancelling pending tasks and resetting stuck ones needs backend
+knowledge; an adapter ships for `django-tasks-db` and the interface is open for others.
+
+## Requirements
+
+- Python 3.12+
+- Django 6.0 or 6.1 (tested against `main` too)
+- A `django.tasks` backend. The reference setup is `django-tasks-db` (`pip install
+  "django-overseer[db]"`).
+
+## Install
+
+```bash
+pip install "django-overseer[db]"
+```
+
+```python
+# settings.py
+INSTALLED_APPS = [
+    ...
+    "django_tasks_db",
+    "overseer",
+]
+
+TASKS = {
+    "default": {
+        "BACKEND": "django_tasks_db.backend.DatabaseBackend",
+        "QUEUES": ["default", "emails"],
+    }
+}
+```
+
+```python
+# urls.py
+from django.urls import include, path
+
+urlpatterns = [
+    path("admin/", admin.site.urls),
+    path("overseer/", include("overseer.urls")),
+]
+```
+
+```bash
+python manage.py migrate
+```
+
+Then run the processes you need:
+
+```bash
+python manage.py overseer_worker --queue-name='*'   # a worker with heartbeats
+python manage.py overseer_scheduler                 # schedules, rescue, metrics, alerts
+```
+
+Open `/overseer/` as a staff user with the `overseer.view_dashboard` permission (superusers
+always have it). `overseer.manage_jobs` allows retry / cancel / dismiss and
+`overseer.manage_schedules` allows pausing and triggering schedules.
+
+## Declaring tasks
+
+`overseer.task` is `django.tasks.task` plus a policy. It accepts the same arguments
+(`priority`, `queue_name`, `backend`, `takes_context`) and returns a normal `Task`, so
+`.enqueue()`, `.using()`, `.call()` and `.get_result()` all work as documented by Django.
+
+```python
+import overseer
+from django.tasks import task
+
+
+@overseer.task(retries=3, backoff="exponential", backoff_base=30, backoff_max=600)
+def charge_card(order_id):
+    ...
+
+
+@overseer.task(retries=5, retry_on=(ConnectionError, TimeoutError), timeout=120)
+def sync_crm(account_id):
+    ...
+
+
+@overseer.task(unique=True, queue_name="emails", tags=("mail",))
+def send_receipt(email):
+    ...
+
+
+@overseer.task(unique=lambda report_id, **kwargs: f"report:{report_id}")
+def build_report(report_id, force=False):
+    ...
+
+
+@overseer.schedule("0 2 * * *", name="nightly-cleanup", timezone="Europe/Paris")
+@overseer.task(retries=1)
+def nightly_cleanup():
+    ...
+
+
+@overseer.schedule(every=300)
+@task  # schedules work on plain django.tasks tasks as well
+def refresh_rates():
+    ...
+```
+
+| Argument | Meaning |
+| --- | --- |
+| `retries` | attempts *after* the first (default `OVERSEER_DEFAULT_RETRIES`, 0) |
+| `backoff` | `"exponential"` (default), `"linear"` or `"constant"` |
+| `backoff_base` / `backoff_max` | seconds; the delay before attempt *n* is `base * 2**(n-2)` capped at `max` for exponential, `base * (n-1)` for linear, `base` for constant |
+| `jitter` | multiply the delay by a random factor between 0.8 and 1.2 (default on) |
+| `retry_on` | tuple of exception classes that trigger a retry (default: any `Exception`) |
+| `timeout` | seconds after which a running attempt is treated as abandoned and rescued |
+| `unique` | `True` (key from task path + arguments), a string, or a callable receiving the task arguments |
+| `tags` | labels shown in the dashboard |
+
+Tasks declared with plain `django.tasks.task` are recorded too, with the
+`OVERSEER_DEFAULT_*` policy. Task modules named `tasks.py` in installed apps are imported at
+startup (`OVERSEER_AUTODISCOVER`), so schedules and policies exist in every process.
+
+Policy only ever applies on failure. Overseer never changes what your task does, when it
+runs, or what the backend returns.
+
+## Schedules
+
+`overseer.schedule` registers a declaration. `overseer_scheduler` (or `python manage.py
+overseer_sync_schedules`) writes it to the `Schedule` table, from which the dashboard can
+pause, resume or trigger it. Rows created in the dashboard or admin, with
+`declared_in_code=False`, are never touched by sync. A declaration removed from the code
+disables its row rather than deleting it, so its history stays.
+
+Cron expressions are the standard five fields with ranges, steps, lists, month and weekday
+names, `@hourly`-style aliases and the usual "day-of-month OR day-of-week" rule. The
+expression is evaluated in the schedule's `timezone`, else `OVERSEER_SCHEDULER_TIMEZONE`,
+else `TIME_ZONE`. A schedule that was missed while no scheduler was running fires once and
+continues from now; missed occurrences are not replayed.
+
+The scheduler loop also runs the rescue every `OVERSEER_RESCUE_INTERVAL` seconds and the
+metrics rollup and alert evaluation every `OVERSEER_MAINTENANCE_INTERVAL` seconds. Run
+several schedulers for availability; schedules are claimed with row locks, so each fires
+once.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `overseer_worker` | `db_worker` with `--heartbeat` seconds (default 10) and a `Worker` row. Accepts every `db_worker` option (`--queue-name`, `--exclude-queues`, `--interval`, `--batch`, `--max-tasks`, `--worker-id`, `--backend`, `--no-startup-delay`, `--reload`). SIGTERM/SIGINT finishes the current task then exits. |
+| `overseer_scheduler [--interval S] [--once]` | syncs schedules, fires what is due, rescues abandoned runs, rolls up metrics, evaluates alerts |
+| `overseer_sync_schedules` | one-off sync of code declarations into the `Schedule` table |
+| `overseer_rescue` | one-off pass over running attempts that exceeded their timeout |
+| `overseer_rollup [--since ISO]` | recompute per-minute metric buckets |
+| `overseer_alerts` | one-off alert evaluation and notification |
+| `overseer_prune [--days N] [--metrics-days N]` | delete finished jobs, runs, alerts, metric buckets and silent workers older than the retention |
+
+If you run `db_worker` directly instead of `overseer_worker`, everything still works: the
+worker shows up in the dashboard from the task signals, only the heartbeat and the host /
+pid details are missing.
+
+## Dashboard
+
+| Page | Content |
+| --- | --- |
+| Overview | throughput, failure rate, runtimes, waiting / running / scheduled counts, workers online, runs-per-minute chart, queue table, recent failures, open alerts |
+| Queues | per queue: waiting, running, processed, failed, failure rate, average and p95 runtime, average wait and the backend's own depth when the adapter supports it |
+| Tasks | the same per task path, with the declared policy |
+| Jobs | filter by status, queue, task, worker; free-text search over task path, job id, result id and unique key |
+| Job | arguments, policy, every attempt with timing, worker, exception and traceback, return value; retry / cancel / dismiss |
+| Failed | open failures with the last error; retry all / dismiss all |
+| Schedules | next and last run, run count, pause / resume / run now / sync |
+| Workers | host, pid, queues, last heartbeat, current run, counters; offline after `OVERSEER_WORKER_OFFLINE_AFTER` |
+| Metrics | per-minute runs and p95 runtime for the last 15 minutes to 24 hours, per queue |
+| Alerts | open and resolved alerts |
+
+Every page accepts `?minutes=15|60|360|1440`. Panels refresh every
+`OVERSEER_REFRESH_SECONDS` seconds (there is a pause button). The same data is available as
+JSON under `/overseer/api/...` for the logged-in user, plus `/overseer/api/health/`, which
+returns HTTP 503 when tasks are waiting and no worker is online.
+
+## Alerts
+
+`overseer_scheduler` (or `overseer_alerts`) evaluates four conditions over the last
+`OVERSEER_ALERT_WINDOW_MINUTES` minutes:
+
+| Kind | Fires when |
+| --- | --- |
+| `failure_rate` | a queue's failure rate exceeds `OVERSEER_ALERT_FAILURE_RATE` (after at least 5 finished runs) |
+| `queue_wait` | the oldest waiting run has waited more than `OVERSEER_ALERT_WAIT_SECONDS` |
+| `queue_depth` | more than `OVERSEER_ALERT_QUEUE_DEPTH` runs are waiting |
+| `worker_offline` | a worker that did not stop cleanly has not been seen for `OVERSEER_WORKER_OFFLINE_AFTER` seconds |
+
+An alert fires once, then not again for `OVERSEER_ALERT_COOLDOWN_MINUTES`, and resolves
+itself when the condition clears. Notifications go to `OVERSEER_ALERT_EMAILS` (or `ADMINS` when that is empty),
+`OVERSEER_SLACK_WEBHOOK_URL` and every callable in `OVERSEER_NOTIFIERS` (`callable(alert)`);
+a failing notifier is logged, never raised. The signals `overseer.signals.alert_fired`,
+`alert_resolved` and `job_failed` (`job`, `run`) let you hook anything else.
+
+## Settings
+
+All settings are optional. See [docs/settings.md](docs/settings.md) for the full reference.
+
+```python
+OVERSEER_DEFAULT_RETRIES = 0           # policy for tasks not declared with overseer.task
+OVERSEER_DEFAULT_BACKOFF = "exponential"
+OVERSEER_DEFAULT_BACKOFF_BASE = 30.0
+OVERSEER_DEFAULT_BACKOFF_MAX = 3600.0
+OVERSEER_DEFAULT_JITTER = True
+OVERSEER_DEFAULT_TIMEOUT = None
+OVERSEER_STALE_AFTER = 3600            # a running attempt with no timeout is abandoned after this
+OVERSEER_WORKER_OFFLINE_AFTER = 120
+OVERSEER_RETENTION_DAYS = 14
+OVERSEER_METRICS_RETENTION_DAYS = 30
+OVERSEER_RECORD_ARGS = True            # False to keep task arguments out of the database
+OVERSEER_MAX_TRACEBACK_CHARS = 20_000
+OVERSEER_PERMISSION = "overseer.view_dashboard"   # None: any staff user
+OVERSEER_REFRESH_SECONDS = 5
+OVERSEER_ALERT_WINDOW_MINUTES = 5
+OVERSEER_ALERT_FAILURE_RATE = 0.25
+OVERSEER_ALERT_WAIT_SECONDS = 60
+OVERSEER_ALERT_QUEUE_DEPTH = 1000
+OVERSEER_ALERT_COOLDOWN_MINUTES = 15
+OVERSEER_NOTIFIERS = []                # dotted paths or callables taking an Alert
+OVERSEER_ALERT_EMAILS = []
+OVERSEER_SLACK_WEBHOOK_URL = None
+OVERSEER_SCHEDULER_INTERVAL = 1.0
+OVERSEER_RESCUE_INTERVAL = 30.0
+OVERSEER_MAINTENANCE_INTERVAL = 60.0
+OVERSEER_SCHEDULER_TIMEZONE = None     # None: TIME_ZONE
+OVERSEER_AUTODISCOVER = True           # import <app>.tasks for every installed app
+OVERSEER_TASK_MODULES = []             # extra modules to import at startup
+```
+
+`python manage.py check` warns when a task declares retries on a backend that cannot defer
+(`supports_defer` is false): retries then run immediately instead of after the backoff.
+
+## How it works
+
+See [docs/design.md](docs/design.md). In short: `task_enqueued`, `task_started` and
+`task_finished` create and update `Job` (one logical unit of work), `Run` (one attempt) and
+`Worker` rows. Failure handling happens in the `task_finished` receiver, inside the worker
+process, so a retry is enqueued by the same process that saw the failure and is visible
+immediately. Recorders never raise: a bug in Overseer cannot break your worker.
+
+## Production notes
+
+- Run `overseer_prune` daily (a schedule works: `@overseer.schedule("0 3 * * *")` on a task
+  that calls `overseer.prune.prune()`), or the tables grow forever.
+- Put the dashboard behind your usual staff authentication; it is a normal Django app under
+  your `LOGIN_URL`.
+- Set `OVERSEER_RECORD_ARGS = False` if task arguments may contain secrets or personal
+  data; tracebacks are still stored, capped at `OVERSEER_MAX_TRACEBACK_CHARS`.
+- On SQLite with several processes (workers, the scheduler, the web app) writing to one
+  file, configure the database as Django recommends for concurrent writers:
+  `"OPTIONS": {"transaction_mode": "IMMEDIATE", "timeout": 20, "init_command": "PRAGMA
+  journal_mode=WAL;"}`. Without it a writer that already read in the same transaction can
+  fail with "database is locked" instead of waiting.
+- `ENQUEUE_ON_COMMIT` is honoured: a task enqueued inside a transaction is recorded when the
+  transaction commits, and never if it rolls back.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest                                         # SQLite, in memory
+OVERSEER_SQLITE_FILE=/tmp/o.sqlite3 pytest tests/test_integration.py   # real subprocesses
+pytest --ds=tests.settings_postgres           # everything, including the concurrency tests
+ruff check src tests && ruff format --check src tests
+```
+
+The suite has four layers: unit tests for every module, integration tests that spawn real
+`overseer_worker` and `overseer_scheduler` processes against a shared database,
+end-to-end tests that drive the dashboard over HTTP with `requests`, and performance
+guards that assert flat query counts and time budgets with thousands of rows.
+
+## License
+
+MIT