percival 0.1.0__tar.gz

percival-0.1.0/.gitignore

@@ -0,0 +1,30 @@
+# python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# node
+node_modules/
+web/dist/
+*.tsbuildinfo
+
+# local stuff
+.env
+*.sqlite3
+*.db
+.idea/
+.vscode/
+.DS_Store
+
+# built ui, produced by `npm run build` in web/ (and by the docker build)
+server/percival_server/static/

percival-0.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.5
+Name: percival
+Version: 0.1.0
+Summary: Client hooks for the Percival Celery control panel: run events, worker versions, stale-worker gate, duplicate guard.
+Project-URL: Homepage, https://github.com/RijinRaju/percival
+Project-URL: Source, https://github.com/RijinRaju/percival
+Project-URL: Issues, https://github.com/RijinRaju/percival/issues
+Author: Rijin Raju
+License-Expression: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Celery
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: celery-redbeat>=2.0
+Requires-Dist: celery>=5.3
+Requires-Dist: redis>=4.5
+Provides-Extra: dev
+Requires-Dist: fakeredis>=2.20; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: import
+Description-Content-Type: text/markdown
+
+# percival
+
+The client half of [Percival](https://github.com/RijinRaju/percival), a
+self-hosted control panel for Celery. You install it in your Celery project.
+It lets the panel change schedules without restarting beat or workers, and it
+stops two things a ping monitor can't see: a task running twice, and a stale
+worker running old code.
+
+Needs Celery 5.3+ and Redis. The broker can be Redis or RabbitMQ.
+
+## Setup
+
+```
+pip install percival
+```
+
+```python
+# proj/celery.py, right after app = Celery(...)
+import os, percival
+
+percival.init(
+    app,
+    redis_url="redis://localhost:6379/0",
+    code_version=os.environ.get("GIT_SHA", "dev"),  # whatever identifies a deploy
+)
+```
+
+```
+celery -A proj beat -S percival.beat.Scheduler
+```
+
+Then run the panel, one container:
+
+```
+docker run -d -p 8080:8080 -v percival:/data \
+  -e PERCIVAL_REDIS_URL=redis://host.docker.internal:6379/0 \
+  -e PERCIVAL_ADMIN_PASSWORD=pick-a-long-one \
+  ghcr.io/rijinraju/percival
+```
+
+Or install it with `pip install percival-server`.
+
+## What `init()` sets up
+
+- **Run events.** Started, success, failure, retry and revoked go to a Redis
+  stream, with timing, worker, code version and redacted args. Anything named
+  like `password`, `token` or `secret` is replaced before it leaves the worker.
+- **Worker heartbeat.** Every worker reports its hostname, queues and code
+  version every 30 seconds.
+- **Task metadata.** The signature, docstring and source of each task, so the
+  panel can show them. Pass `publish_source=False` to keep source private.
+- **Stale worker gate.** A schedule can require a code version. A worker on a
+  different version doesn't run the task. The message goes back on the queue
+  for an up-to-date worker.
+
+## Stopping duplicate runs
+
+```python
+from percival import once_per
+
+
+@app.task
+@once_per(key=lambda user_id, **kw: user_id, window="1d")
+def send_digest(user_id): ...
+```
+
+A second run for the same key in the same window is skipped and recorded.
+
+## Moving an existing `beat_schedule`
+
+```
+percival import --app proj.celery:app --redis-url redis://localhost:6379/0 --apply
+```
+
+Then remove those entries from `beat_schedule`.
+
+The [full README](https://github.com/RijinRaju/percival#readme) has every
+option and a `docker compose up` demo.

percival-0.1.0/README.md

@@ -0,0 +1,78 @@
+# percival
+
+The client half of [Percival](https://github.com/RijinRaju/percival), a
+self-hosted control panel for Celery. You install it in your Celery project.
+It lets the panel change schedules without restarting beat or workers, and it
+stops two things a ping monitor can't see: a task running twice, and a stale
+worker running old code.
+
+Needs Celery 5.3+ and Redis. The broker can be Redis or RabbitMQ.
+
+## Setup
+
+```
+pip install percival
+```
+
+```python
+# proj/celery.py, right after app = Celery(...)
+import os, percival
+
+percival.init(
+    app,
+    redis_url="redis://localhost:6379/0",
+    code_version=os.environ.get("GIT_SHA", "dev"),  # whatever identifies a deploy
+)
+```
+
+```
+celery -A proj beat -S percival.beat.Scheduler
+```
+
+Then run the panel, one container:
+
+```
+docker run -d -p 8080:8080 -v percival:/data \
+  -e PERCIVAL_REDIS_URL=redis://host.docker.internal:6379/0 \
+  -e PERCIVAL_ADMIN_PASSWORD=pick-a-long-one \
+  ghcr.io/rijinraju/percival
+```
+
+Or install it with `pip install percival-server`.
+
+## What `init()` sets up
+
+- **Run events.** Started, success, failure, retry and revoked go to a Redis
+  stream, with timing, worker, code version and redacted args. Anything named
+  like `password`, `token` or `secret` is replaced before it leaves the worker.
+- **Worker heartbeat.** Every worker reports its hostname, queues and code
+  version every 30 seconds.
+- **Task metadata.** The signature, docstring and source of each task, so the
+  panel can show them. Pass `publish_source=False` to keep source private.
+- **Stale worker gate.** A schedule can require a code version. A worker on a
+  different version doesn't run the task. The message goes back on the queue
+  for an up-to-date worker.
+
+## Stopping duplicate runs
+
+```python
+from percival import once_per
+
+
+@app.task
+@once_per(key=lambda user_id, **kw: user_id, window="1d")
+def send_digest(user_id): ...
+```
+
+A second run for the same key in the same window is skipped and recorded.
+
+## Moving an existing `beat_schedule`
+
+```
+percival import --app proj.celery:app --redis-url redis://localhost:6379/0 --apply
+```
+
+Then remove those entries from `beat_schedule`.
+
+The [full README](https://github.com/RijinRaju/percival#readme) has every
+option and a `docker compose up` demo.

percival-0.1.0/percival/__init__.py

@@ -0,0 +1,67 @@
+"""Client side of Percival. Installed in the Celery project, not in the panel.
+
+import percival
+percival.init(app, redis_url="redis://...", code_version=os.environ["GIT_SHA"])
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any
+
+__version__ = "0.1.0"
+
+from percival.config import UNKNOWN_VERSION, Settings
+from percival.dedupe import once_per
+from percival.runtime import ClientFactory, Runtime
+
+log = logging.getLogger(__name__)
+
+__all__ = ["Settings", "__version__", "init", "once_per", "shutdown"]
+
+
+def init(
+    app: Any,
+    redis_url: str | None = None,
+    code_version: str | None = None,
+    *,
+    client_factory: ClientFactory | None = None,
+    **options: Any,
+) -> Runtime:
+    """Wire percival into a celery app. Safe to call more than once.
+
+    Any field of :class:`percival.Settings` can be passed as a keyword, e.g.
+    ``redact_kwargs=[...]``, ``stale_action="drop"``, ``capture_args=False``.
+    ``client_factory`` exists for tests, you shouldn't need it.
+    """
+    # imported here so `import percival` stays cheap for things like the CLI
+    from percival import gate, heartbeat, metadata, runtime, signals
+
+    settings = Settings.build(redis_url=redis_url, code_version=code_version, **options)
+    if settings.code_version == UNKNOWN_VERSION:
+        log.warning(
+            "percival: no code_version set. pass code_version= or set PERCIVAL_CODE_VERSION, "
+            "otherwise schedules that require a version will be blocked on this worker"
+        )
+
+    rt = runtime.configure(settings, client_factory=client_factory)
+    signals.connect()
+    heartbeat.connect()
+    metadata.connect()
+    gate.connect()
+
+    # if the worker is already up (init called late), gate the tasks now
+    if getattr(app, "finalized", False):
+        gate.install(app)
+    return rt
+
+
+def shutdown() -> None:
+    """Undo init(). Mostly useful in tests."""
+    from percival import gate, heartbeat, metadata, runtime, signals
+
+    signals.disconnect()
+    heartbeat.disconnect()
+    metadata.disconnect()
+    gate.disconnect()
+    runtime.reset()

percival-0.1.0/percival/beat.py

@@ -0,0 +1,91 @@
+"""RedBeat scheduler with per-schedule timezones and a heartbeat.
+
+    celery -A proj beat -S percival.beat.Scheduler
+
+It's stock RedBeat underneath. Entries stay in RedBeat's format, so plain
+``redbeat.RedBeatScheduler`` still reads them (crontabs just fall back to the
+app timezone). What this adds:
+
+- crontabs with a ``percival_timezone`` header are evaluated in that zone
+- a heartbeat key so the panel knows beat is alive, when it last ticked and
+  who holds the lock
+- a shorter default loop interval, so a newly created schedule is picked up
+  in seconds instead of up to 5 minutes
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import socket
+import time
+from typing import Any
+
+from celery.schedules import crontab
+from redbeat.schedulers import RedBeatScheduler, RedBeatSchedulerEntry
+
+from percival import __version__, runtime
+from percival.naming import Headers
+from percival.schedules import zoned_crontab
+
+log = logging.getLogger(__name__)
+
+DEFAULT_MAX_INTERVAL = 15
+
+
+def apply_zone(sched: Any, options: dict[str, Any] | None, app: Any = None) -> Any:
+    headers = (options or {}).get("headers") or {}
+    zone = headers.get(Headers.TIMEZONE)
+    if not zone or not isinstance(sched, crontab) or isinstance(sched, zoned_crontab):
+        return sched
+    try:
+        return zoned_crontab.from_crontab(sched, zone, app=app)
+    except Exception:
+        log.warning("percival: unknown timezone %r, using the app timezone", zone)
+        return sched
+
+
+class Entry(RedBeatSchedulerEntry):  # type: ignore[misc]
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        super().__init__(*args, **kwargs)
+        self.schedule = apply_zone(self.schedule, self.options, self.app)  # type: ignore[has-type]
+
+
+class Scheduler(RedBeatScheduler):  # type: ignore[misc]
+    Entry = Entry
+
+    def __init__(self, app: Any, *args: Any, **kwargs: Any) -> None:
+        if kwargs.get("max_interval") is None and not app.conf.beat_max_loop_interval:
+            kwargs["max_interval"] = DEFAULT_MAX_INTERVAL
+        super().__init__(app, *args, **kwargs)
+        self._started_at = time.time()
+
+    def tick(self, *args: Any, **kwargs: Any) -> Any:
+        result = super().tick(*args, **kwargs)
+        self._heartbeat()
+        return result
+
+    def _heartbeat(self) -> None:
+        rt = runtime.get_optional()
+        if rt is None:
+            return
+        payload = {
+            "hostname": socket.gethostname(),
+            "pid": os.getpid(),
+            "scheduler": f"{type(self).__module__}.{type(self).__name__}",
+            "started_at": self._started_at,
+            "last_tick": time.time(),
+            "max_interval": self.max_interval,
+            "lock_key": self.lock_key,
+            "code_version": rt.settings.code_version,
+            "client_version": __version__,
+        }
+        try:
+            rt.redis.get().set(
+                rt.keys.beat,
+                json.dumps(payload),
+                ex=int(max(self.max_interval * 3, 60)),
+            )
+        except Exception:
+            log.debug("percival: beat heartbeat failed", exc_info=True)

percival-0.1.0/percival/cli.py

@@ -0,0 +1,78 @@
+"""``percival`` command line. Only ``import`` for now."""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import os
+import sys
+from typing import Any
+
+from percival.naming import NAME
+
+
+def load_app(spec: str) -> Any:
+    """``proj.celery:app``, or just ``proj.celery`` if the app is called app/celery."""
+    # same as celery -A: make the current directory importable
+    if os.getcwd() not in sys.path:
+        sys.path.insert(0, os.getcwd())
+
+    module_name, _, attr = spec.partition(":")
+    module = importlib.import_module(module_name)
+    if attr:
+        return getattr(module, attr)
+    for candidate in ("app", "celery"):
+        if hasattr(module, candidate):
+            return getattr(module, candidate)
+    raise SystemExit(f"couldn't find a celery app in {module_name}, try {module_name}:app")
+
+
+def cmd_import(args: argparse.Namespace) -> int:
+    from percival import importer
+
+    app = load_app(args.app)
+    importer.configure_redbeat(app, args.redis_url, args.key_prefix)
+    planned = importer.plan(app, require_version=args.require_version, only=args.only or None)
+    print(importer.render(planned))
+
+    if not args.apply:
+        if any(p.action in (importer.CREATE, importer.UPDATE) for p in planned):
+            print("\ndry run, nothing written. run again with --apply to write these.")
+        return 0
+
+    written = importer.apply(app, planned)
+    print(f"\nwrote {written} entries to redbeat.")
+    if written:
+        print(
+            "note: beat re-syncs beat_schedule from config on every start, so remove the\n"
+            "imported entries from your config or they will overwrite changes made in the panel."
+        )
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog=NAME)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    imp = sub.add_parser("import", help="copy beat_schedule from your app into redbeat")
+    imp.add_argument("--app", required=True, help="celery app, e.g. proj.celery:app")
+    imp.add_argument("--redis-url", help="redis used by redbeat (default: redbeat_redis_url)")
+    imp.add_argument("--key-prefix", help="redbeat_key_prefix if you changed it")
+    imp.add_argument(
+        "--require-version",
+        help="set the required code version on every imported entry",
+    )
+    imp.add_argument("--only", nargs="+", metavar="NAME", help="import just these entries")
+    imp.add_argument("--apply", action="store_true", help="actually write (default is dry run)")
+    imp.set_defaults(func=cmd_import)
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    result: int = args.func(args)
+    return result
+
+
+if __name__ == "__main__":
+    sys.exit(main())

percival-0.1.0/percival/config.py

@@ -0,0 +1,102 @@
+from __future__ import annotations
+
+import os
+from collections.abc import Iterable
+from dataclasses import dataclass, replace
+from typing import Any, Literal
+
+from percival.naming import ENV_PREFIX, NAME
+
+DEFAULT_REDACT = (
+    "password",
+    "passwd",
+    "secret",
+    "token",
+    "api_key",
+    "apikey",
+    "authorization",
+    "cookie",
+)
+
+UNKNOWN_VERSION = "unknown"
+
+StaleAction = Literal["requeue", "drop"]
+
+
+def _env(name: str) -> str | None:
+    value = os.environ.get(ENV_PREFIX + name)
+    return value if value else None
+
+
+def _env_bool(name: str) -> bool | None:
+    value = _env(name)
+    if value is None:
+        return None
+    return value.strip().lower() in {"1", "true", "yes", "on"}
+
+
+@dataclass(frozen=True)
+class Settings:
+    redis_url: str = "redis://localhost:6379/0"
+    code_version: str = UNKNOWN_VERSION
+    key_prefix: str = NAME
+
+    # what goes into run events
+    capture_args: bool = True
+    redact_kwargs: tuple[str, ...] = DEFAULT_REDACT
+    max_value_chars: int = 2_000
+    max_traceback_chars: int = 20_000
+
+    # what the stale-worker gate does with a blocked message
+    stale_action: StaleAction = "requeue"
+    requeue_delay: float = 5.0
+    max_requeues: int = 50
+
+    heartbeat_interval: float = 30.0
+    heartbeat_ttl: int = 90
+
+    events_maxlen: int = 100_000
+    publish_source: bool = True
+    source_max_lines: int = 300
+
+    # once_per behaviour when redis is down; False means skip the run
+    dedupe_fail_open: bool = False
+
+    @classmethod
+    def build(cls, **overrides: Any) -> Settings:
+        """Explicit arguments win, then PERCIVAL_* env vars, then defaults."""
+        values: dict[str, Any] = {}
+
+        env_map: dict[str, Any] = {
+            "redis_url": _env("REDIS_URL"),
+            "code_version": _env("CODE_VERSION"),
+            "key_prefix": _env("KEY_PREFIX"),
+            "capture_args": _env_bool("CAPTURE_ARGS"),
+            "dedupe_fail_open": _env_bool("DEDUPE_FAIL_OPEN"),
+            "stale_action": _env("STALE_ACTION"),
+        }
+        values.update({k: v for k, v in env_map.items() if v is not None})
+        values.update({k: v for k, v in overrides.items() if v is not None})
+
+        redact = values.get("redact_kwargs")
+        if redact is not None and not isinstance(redact, tuple):
+            values["redact_kwargs"] = _as_tuple(redact)
+
+        settings = cls(**values)
+        settings.validate()
+        return settings
+
+    def validate(self) -> None:
+        if self.stale_action not in ("requeue", "drop"):
+            raise ValueError(f"stale_action must be 'requeue' or 'drop', got {self.stale_action!r}")
+        if self.heartbeat_ttl <= self.heartbeat_interval:
+            raise ValueError("heartbeat_ttl has to be larger than heartbeat_interval")
+
+    def with_(self, **changes: Any) -> Settings:
+        return replace(self, **changes)
+
+
+def _as_tuple(items: Iterable[str] | str) -> tuple[str, ...]:
+    if isinstance(items, str):
+        return tuple(s.strip() for s in items.split(",") if s.strip())
+    return tuple(items)