pybgworker 0.2.1__tar.gz → 0.2.2__tar.gz

{pybgworker-0.2.1 → pybgworker-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pybgworker
-Version: 0.2.1
+Version: 0.2.2
 Summary: Lightweight production-ready background task worker with cron, rate limiting and JSON observability
 Author: Prabhat Verma
 License: MIT
@@ -25,33 +25,38 @@ Dynamic: license-file
 
 # PyBgWorker
 
-A lightweight, production-ready background task framework for Python.
+A lightweight, production-ready background task library for Python.
 
-PyBgWorker provides a durable SQLite-backed task queue, cron scheduling,
-rate limiting, retries, and structured observability — all without external
-infrastructure.
+PyBgWorker provides a durable SQLite-backed task queue, scheduling (cron and
+countdown/ETA), rate limiting, retries, and structured observability without
+external infrastructure.
 
 It is designed to be simple, reliable, and easy to deploy.
 
 ---
 
-## ✨ Features
+## Features
 
 - Persistent SQLite task queue
 - Multi-worker safe execution
+- Task scheduling: cron + countdown/ETA
 - Retry + failure handling
+- Task cancellation support
 - Crash isolation via subprocess
-- Cron scheduler for recurring jobs
+- Task priority execution
+- Task status tracking
+- Result storage and retrieval
+- Worker statistics and monitoring
 - JSON structured logging
 - Task duration tracking
 - Rate limiting (overload protection)
 - Heartbeat monitoring
-- CLI inspect / retry / purge / cancel
+- CLI tools: inspect, retry, failed, purge, cancel, stats
 - Production-safe worker loop
 
 ---
 
-## 🚀 Installation
+## Installation
 
 ```bash
 pip install pybgworker
@@ -59,7 +64,7 @@ pip install pybgworker
 
 ---
 
-## 🧠 Basic Usage
+## Basic Usage
 
 ### Define a task
 
@@ -79,7 +84,7 @@ add.delay(1, 2)
 
 ---
 
-## ▶ Run worker
+## Run worker
 
 ```bash
 python -m pybgworker.cli run --app example
@@ -87,7 +92,7 @@ python -m pybgworker.cli run --app example
 
 ---
 
-## ⏰ Cron Scheduler
+## Cron Scheduler
 
 Run recurring tasks:
 
@@ -105,7 +110,7 @@ Cron runs automatically inside the worker.
 
 ---
 
-## 📊 JSON Logging
+## JSON Logging
 
 All worker events are structured JSON:
 
@@ -123,7 +128,7 @@ This enables:
 
 ---
 
-## 🚦 Rate Limiting
+## Rate Limiting
 
 Protect infrastructure from overload:
 
@@ -135,7 +140,7 @@ Ensures predictable execution under heavy load.
 
 ---
 
-## 🔍 CLI Commands
+## CLI Commands
 
 Inspect queue:
 
@@ -163,7 +168,7 @@ python -m pybgworker.cli purge
 
 ---
 
-## 🧪 Observability
+## Observability
 
 PyBgWorker logs:
 
@@ -181,7 +186,7 @@ All machine-readable.
 
 ---
 
-## 🎯 Design Goals
+## Design Goals
 
 - zero external dependencies
 - SQLite durability
@@ -192,18 +197,23 @@ All machine-readable.
 
 ---
 
-## 📌 Roadmap
+## Roadmap
 
-Future upgrades may include:
+Planned but not yet included:
 
-- dashboard web UI
-- metrics endpoint
-- Redis backend
-- workflow pipelines
-- cluster coordination
+- Single-worker concurrency (process pool)
+- Retry backoff + jitter policies
+- Dead-letter queue for exhausted retries
+- Task/result TTL and automatic DB cleanup
+- Multiple named queues + routing
+- Pluggable backends (Redis first)
+- Cluster coordination / leader election for scheduler
+- Metrics endpoint and health checks
+- Dashboard API + web UI
+- Workflow pipelines / DAGs
 
 ---
 
-## 📄 License
+## License
 
 MIT License

{pybgworker-0.2.1 → pybgworker-0.2.2}/README.md

@@ -1,184 +1,194 @@
-# PyBgWorker
-
-A lightweight, production-ready background task framework for Python.
-
-PyBgWorker provides a durable SQLite-backed task queue, cron scheduling,
-rate limiting, retries, and structured observability — all without external
-infrastructure.
-
-It is designed to be simple, reliable, and easy to deploy.
-
----
-
-## ✨ Features
-
-- Persistent SQLite task queue
-- Multi-worker safe execution
-- Retry + failure handling
-- Crash isolation via subprocess
-- Cron scheduler for recurring jobs
-- JSON structured logging
-- Task duration tracking
-- Rate limiting (overload protection)
-- Heartbeat monitoring
-- CLI inspect / retry / purge / cancel
-- Production-safe worker loop
-
----
-
-## 🚀 Installation
-
-```bash
-pip install pybgworker
-```
-
----
-
-## 🧠 Basic Usage
-
-### Define a task
-
-```python
-from pybgworker.task import task
-
-@task(name="add")
-def add(a, b):
-    return a + b
-```
-
-### Enqueue a task
-
-```python
-add.delay(1, 2)
-```
-
----
-
-## ▶ Run worker
-
-```bash
-python -m pybgworker.cli run --app example
-```
-
----
-
-## ⏰ Cron Scheduler
-
-Run recurring tasks:
-
-```python
-from pybgworker.scheduler import cron
-from pybgworker.task import task
-
-@task(name="heartbeat_task")
-@cron("*/1 * * * *")
-def heartbeat():
-    print("alive")
-```
-
-Cron runs automatically inside the worker.
-
----
-
-## 📊 JSON Logging
-
-All worker events are structured JSON:
-
-```json
-{"event":"task_start","task_id":"..."}
-{"event":"task_success","duration":0.12}
-```
-
-This enables:
-
-- monitoring
-- analytics
-- alerting
-- observability pipelines
-
----
-
-## 🚦 Rate Limiting
-
-Protect infrastructure from overload:
-
-```python
-RATE_LIMIT = 5  # tasks per second
-```
-
-Ensures predictable execution under heavy load.
-
----
-
-## 🔍 CLI Commands
-
-Inspect queue:
-
-```bash
-python -m pybgworker.cli inspect
-```
-
-Retry failed task:
-
-```bash
-python -m pybgworker.cli retry <task_id>
-```
-
-Cancel task:
-
-```bash
-python -m pybgworker.cli cancel <task_id>
-```
-
-Purge queued tasks:
-
-```bash
-python -m pybgworker.cli purge
-```
-
----
-
-## 🧪 Observability
-
-PyBgWorker logs:
-
-- worker start
-- cron events
-- task start
-- success
-- retry
-- failure
-- timeout
-- crash
-- heartbeat errors
-
-All machine-readable.
-
----
-
-## 🎯 Design Goals
-
-- zero external dependencies
-- SQLite durability
-- safe multiprocessing
-- operator-friendly CLI
-- production observability
-- infrastructure protection
-
----
-
-## 📌 Roadmap
-
-Future upgrades may include:
-
-- dashboard web UI
-- metrics endpoint
-- Redis backend
-- workflow pipelines
-- cluster coordination
-
----
-
-## 📄 License
-
-MIT License
+# PyBgWorker
+
+A lightweight, production-ready background task library for Python.
+
+PyBgWorker provides a durable SQLite-backed task queue, scheduling (cron and
+countdown/ETA), rate limiting, retries, and structured observability without
+external infrastructure.
+
+It is designed to be simple, reliable, and easy to deploy.
+
+---
+
+## Features
+
+- Persistent SQLite task queue
+- Multi-worker safe execution
+- Task scheduling: cron + countdown/ETA
+- Retry + failure handling
+- Task cancellation support
+- Crash isolation via subprocess
+- Task priority execution
+- Task status tracking
+- Result storage and retrieval
+- Worker statistics and monitoring
+- JSON structured logging
+- Task duration tracking
+- Rate limiting (overload protection)
+- Heartbeat monitoring
+- CLI tools: inspect, retry, failed, purge, cancel, stats
+- Production-safe worker loop
+
+---
+
+## Installation
+
+```bash
+pip install pybgworker
+```
+
+---
+
+## Basic Usage
+
+### Define a task
+
+```python
+from pybgworker.task import task
+
+@task(name="add")
+def add(a, b):
+    return a + b
+```
+
+### Enqueue a task
+
+```python
+add.delay(1, 2)
+```
+
+---
+
+## Run worker
+
+```bash
+python -m pybgworker.cli run --app example
+```
+
+---
+
+## Cron Scheduler
+
+Run recurring tasks:
+
+```python
+from pybgworker.scheduler import cron
+from pybgworker.task import task
+
+@task(name="heartbeat_task")
+@cron("*/1 * * * *")
+def heartbeat():
+    print("alive")
+```
+
+Cron runs automatically inside the worker.
+
+---
+
+## JSON Logging
+
+All worker events are structured JSON:
+
+```json
+{"event":"task_start","task_id":"..."}
+{"event":"task_success","duration":0.12}
+```
+
+This enables:
+
+- monitoring
+- analytics
+- alerting
+- observability pipelines
+
+---
+
+## Rate Limiting
+
+Protect infrastructure from overload:
+
+```python
+RATE_LIMIT = 5  # tasks per second
+```
+
+Ensures predictable execution under heavy load.
+
+---
+
+## CLI Commands
+
+Inspect queue:
+
+```bash
+python -m pybgworker.cli inspect
+```
+
+Retry failed task:
+
+```bash
+python -m pybgworker.cli retry <task_id>
+```
+
+Cancel task:
+
+```bash
+python -m pybgworker.cli cancel <task_id>
+```
+
+Purge queued tasks:
+
+```bash
+python -m pybgworker.cli purge
+```
+
+---
+
+## Observability
+
+PyBgWorker logs:
+
+- worker start
+- cron events
+- task start
+- success
+- retry
+- failure
+- timeout
+- crash
+- heartbeat errors
+
+All machine-readable.
+
+---
+
+## Design Goals
+
+- zero external dependencies
+- SQLite durability
+- safe multiprocessing
+- operator-friendly CLI
+- production observability
+- infrastructure protection
+
+---
+
+## Roadmap
+
+Planned but not yet included:
+
+- Single-worker concurrency (process pool)
+- Retry backoff + jitter policies
+- Dead-letter queue for exhausted retries
+- Task/result TTL and automatic DB cleanup
+- Multiple named queues + routing
+- Pluggable backends (Redis first)
+- Cluster coordination / leader election for scheduler
+- Metrics endpoint and health checks
+- Dashboard API + web UI
+- Workflow pipelines / DAGs
+
+---
+
+## License
+
+MIT License

{pybgworker-0.2.1 → pybgworker-0.2.2}/pybgworker/__init__.py

@@ -2,5 +2,5 @@ from .task import task
 from .result import AsyncResult
 
 __all__ = ["task", "AsyncResult"]
-__version__ = "0.2.1"
+__version__ = "0.2.2"
 

pybgworker-0.2.2/pybgworker/ratelimit.py

@@ -0,0 +1,43 @@
+import time
+import threading
+
+
+class RateLimiter:
+    def __init__(self, rate_per_sec):
+        # default/global rate
+        self.default_rate = rate_per_sec
+        self.lock = threading.Lock()
+        self.timestamps = []
+
+    def acquire(self, rate=None):
+        """
+        rate: optional per-task rate limit
+        """
+        limit = rate or self.default_rate
+
+        # No limit configured
+        if not limit or limit <= 0:
+            return
+
+        with self.lock:
+            now = time.time()
+
+            # Remove timestamps older than 1 second
+            self.timestamps = [
+                t for t in self.timestamps
+                if now - t < 1
+            ]
+
+            # Wait if limit reached
+            if len(self.timestamps) >= limit:
+                sleep_time = 1 - (now - self.timestamps[0])
+                if sleep_time > 0:
+                    time.sleep(sleep_time)
+
+                now = time.time()
+                self.timestamps = [
+                    t for t in self.timestamps
+                    if now - t < 1
+                ]
+
+            self.timestamps.append(time.time())

{pybgworker-0.2.1 → pybgworker-0.2.2}/pybgworker/sqlite_queue.py

@@ -105,7 +105,9 @@ class SQLiteQueue(BaseQueue):
                 UPDATE tasks
                 SET status='success',
                     finished_at=?,
-                    updated_at=?
+                    updated_at=?,
+                    locked_by=NULL,
+                    locked_at=NULL
                 WHERE id=?
             """, (now().isoformat(), now().isoformat(), task_id))
             conn.commit()
@@ -119,7 +121,9 @@ class SQLiteQueue(BaseQueue):
                 SET status='failed',
                     last_error=?,
                     finished_at=?,
-                    updated_at=?
+                    updated_at=?,
+                    locked_by=NULL,
+                    locked_at=NULL
                 WHERE id=?
             """, (error, now().isoformat(), now().isoformat(), task_id))
             conn.commit()
@@ -134,7 +138,24 @@ class SQLiteQueue(BaseQueue):
                 SET status='retrying',
                     attempt=attempt+1,
                     run_at=?,
-                    updated_at=?
+                    updated_at=?,
+                    locked_by=NULL,
+                    locked_at=NULL
                 WHERE id=?
             """, (run_at.isoformat(), now().isoformat(), task_id))
             conn.commit()
+
+    # ---------------- cancel ----------------
+
+    def cancel(self, task_id):
+        with get_conn() as conn:
+            conn.execute("""
+                UPDATE tasks
+                SET status='cancelled',
+                    finished_at=?,
+                    updated_at=?,
+                    locked_by=NULL,
+                    locked_at=NULL
+                WHERE id=?
+            """, (now().isoformat(), now().isoformat(), task_id))
+            conn.commit()

{pybgworker-0.2.1 → pybgworker-0.2.2}/pybgworker/task.py

@@ -10,17 +10,27 @@ queue = SQLiteQueue()
 backend = SQLiteBackend()
 
 
-def task(name=None, retries=0, retry_delay=0, retry_for=(Exception,)):
+def task(
+    name=None,
+    retries=0,
+    retry_delay=0,
+    retry_for=(Exception,),
+    timeout=None,
+    rate_limit=None,
+):
     if name is None:
         raise ValueError("Task name is required to avoid __main__ issues")
 
     def decorator(func):
         task_name = name or f"{func.__module__}.{func.__name__}"
 
+        # Store task metadata
         TASK_REGISTRY[task_name] = {
             "func": func,
             "retry_delay": retry_delay,
             "retry_for": retry_for,
+            "timeout": timeout,
+            "rate_limit": rate_limit,
         }
 
         @wraps(func)
@@ -42,7 +52,7 @@ def task(name=None, retries=0, retry_delay=0, retry_for=(Exception,)):
                 "attempt": 0,
                 "max_retries": retries,
                 "run_at": run_at.isoformat(),
-                "priority": priority,   # ⭐ NEW
+                "priority": priority,
                 "locked_by": None,
                 "locked_at": None,
                 "last_error": None,

pybgworker-0.2.2/pybgworker/worker.py

@@ -0,0 +1,189 @@
+import time
+import traceback
+import threading
+import signal
+import os
+from multiprocessing import Process, Queue as MPQueue
+
+from .logger import log
+from .sqlite_queue import SQLiteQueue
+from .task import TASK_REGISTRY
+from .config import WORKER_NAME, POLL_INTERVAL, RATE_LIMIT
+from .utils import loads, get_conn, now
+from .backends import SQLiteBackend
+from .scheduler import run_scheduler
+from .ratelimit import RateLimiter
+
+
+queue = SQLiteQueue()
+backend = SQLiteBackend()
+limiter = RateLimiter(RATE_LIMIT)
+
+TASK_TIMEOUT = 150  # default timeout
+
+
+shutdown_requested = False
+last_shutdown_signal = 0
+current_task_id = None
+current_process = None
+
+
+def handle_shutdown(signum, frame):
+    global shutdown_requested, last_shutdown_signal
+    global current_task_id, current_process
+
+    now_ts = time.time()
+
+    # Ignore duplicate signals (Windows issue)
+    if now_ts - last_shutdown_signal < 1:
+        return
+
+    last_shutdown_signal = now_ts
+
+    # Second Ctrl+C → force exit
+    if shutdown_requested:
+        log("worker_force_exit", worker=WORKER_NAME)
+
+        if current_task_id:
+            queue.cancel(current_task_id)
+            log("task_cancelled", task_id=current_task_id)
+
+        if current_process and current_process.is_alive():
+            current_process.terminate()
+
+        os._exit(1)
+
+    shutdown_requested = True
+    log("worker_shutdown_requested", worker=WORKER_NAME)
+
+
+def heartbeat():
+    while True:
+        try:
+            with get_conn() as conn:
+                conn.execute("""
+                    INSERT INTO workers(name, last_seen)
+                    VALUES (?, ?)
+                    ON CONFLICT(name)
+                    DO UPDATE SET last_seen=excluded.last_seen
+                """, (WORKER_NAME, now().isoformat()))
+                conn.commit()
+        except Exception as e:
+            log("heartbeat_error", error=str(e))
+
+        time.sleep(5)
+
+
+def run_task(func, args, kwargs, result_queue):
+    # Child process ignores Ctrl+C
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+    try:
+        result = func(*args, **kwargs)
+        result_queue.put(("success", result))
+    except Exception:
+        result_queue.put(("error", traceback.format_exc()))
+
+
+def run_worker():
+    global shutdown_requested, current_task_id, current_process
+
+    signal.signal(signal.SIGINT, handle_shutdown)
+    signal.signal(signal.SIGTERM, handle_shutdown)
+
+    log("worker_start", worker=WORKER_NAME)
+
+    threading.Thread(target=heartbeat, daemon=True).start()
+    threading.Thread(target=run_scheduler, daemon=True).start()
+
+    while not shutdown_requested:
+        task = queue.fetch_next(WORKER_NAME)
+
+        if not task:
+            if shutdown_requested:
+                break
+            time.sleep(POLL_INTERVAL)
+            continue
+
+        meta = TASK_REGISTRY.get(task["name"])
+        if not meta:
+            queue.fail(task["id"], "Task not registered")
+            log("task_invalid", task_id=task["id"])
+            continue
+
+        # -------- Rate limit per task --------
+        limiter.acquire(meta.get("rate_limit"))
+
+        func = meta["func"]
+        retry_delay = meta["retry_delay"]
+
+        args = loads(task["args"])
+        kwargs = loads(task["kwargs"])
+
+        start_time = now()
+        current_task_id = task["id"]
+
+        log("task_start", task_id=current_task_id, worker=WORKER_NAME)
+
+        result_queue = MPQueue()
+        process = Process(target=run_task, args=(func, args, kwargs, result_queue))
+        current_process = process
+
+        process.start()
+
+        # -------- Timeout per task --------
+        timeout = meta.get("timeout") or TASK_TIMEOUT
+
+        start_join = time.time()
+
+        while process.is_alive():
+            if time.time() - start_join > timeout:
+                break
+            time.sleep(0.2)
+
+        if process.is_alive():
+            process.terminate()
+
+            info = backend.get_task(current_task_id)
+            if info["status"] == "cancelled":
+                log("task_cancelled", task_id=current_task_id)
+                current_task_id = None
+                current_process = None
+                continue
+
+            queue.fail(current_task_id, "Task timeout")
+            log("task_timeout", task_id=current_task_id)
+            log("task_failed", task_id=current_task_id)
+            current_task_id = None
+            current_process = None
+            continue
+
+        if result_queue.empty():
+            queue.fail(current_task_id, "Task crashed")
+            log("task_crash", task_id=current_task_id)
+            current_task_id = None
+            current_process = None
+            continue
+
+        status, payload = result_queue.get()
+        duration = (now() - start_time).total_seconds()
+
+        if status == "success":
+            backend.store_result(current_task_id, payload)
+            queue.ack(current_task_id)
+            log(
+                "task_success",
+                task_id=current_task_id,
+                duration=duration,
+                worker=WORKER_NAME,
+            )
+        else:
+            if task["attempt"] < task["max_retries"]:
+                queue.reschedule(current_task_id, retry_delay)
+            else:
+                queue.fail(current_task_id, payload)
+
+        current_task_id = None
+        current_process = None
+
+    log("worker_stopped", worker=WORKER_NAME)

{pybgworker-0.2.1 → pybgworker-0.2.2}/pybgworker.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pybgworker
-Version: 0.2.1
+Version: 0.2.2
 Summary: Lightweight production-ready background task worker with cron, rate limiting and JSON observability
 Author: Prabhat Verma
 License: MIT
@@ -25,33 +25,38 @@ Dynamic: license-file
 
 # PyBgWorker
 
-A lightweight, production-ready background task framework for Python.
+A lightweight, production-ready background task library for Python.
 
-PyBgWorker provides a durable SQLite-backed task queue, cron scheduling,
-rate limiting, retries, and structured observability — all without external
-infrastructure.
+PyBgWorker provides a durable SQLite-backed task queue, scheduling (cron and
+countdown/ETA), rate limiting, retries, and structured observability without
+external infrastructure.
 
 It is designed to be simple, reliable, and easy to deploy.
 
 ---
 
-## ✨ Features
+## Features
 
 - Persistent SQLite task queue
 - Multi-worker safe execution
+- Task scheduling: cron + countdown/ETA
 - Retry + failure handling
+- Task cancellation support
 - Crash isolation via subprocess
-- Cron scheduler for recurring jobs
+- Task priority execution
+- Task status tracking
+- Result storage and retrieval
+- Worker statistics and monitoring
 - JSON structured logging
 - Task duration tracking
 - Rate limiting (overload protection)
 - Heartbeat monitoring
-- CLI inspect / retry / purge / cancel
+- CLI tools: inspect, retry, failed, purge, cancel, stats
 - Production-safe worker loop
 
 ---
 
-## 🚀 Installation
+## Installation
 
 ```bash
 pip install pybgworker
@@ -59,7 +64,7 @@ pip install pybgworker
 
 ---
 
-## 🧠 Basic Usage
+## Basic Usage
 
 ### Define a task
 
@@ -79,7 +84,7 @@ add.delay(1, 2)
 
 ---
 
-## ▶ Run worker
+## Run worker
 
 ```bash
 python -m pybgworker.cli run --app example
@@ -87,7 +92,7 @@ python -m pybgworker.cli run --app example
 
 ---
 
-## ⏰ Cron Scheduler
+## Cron Scheduler
 
 Run recurring tasks:
 
@@ -105,7 +110,7 @@ Cron runs automatically inside the worker.
 
 ---
 
-## 📊 JSON Logging
+## JSON Logging
 
 All worker events are structured JSON:
 
@@ -123,7 +128,7 @@ This enables:
 
 ---
 
-## 🚦 Rate Limiting
+## Rate Limiting
 
 Protect infrastructure from overload:
 
@@ -135,7 +140,7 @@ Ensures predictable execution under heavy load.
 
 ---
 
-## 🔍 CLI Commands
+## CLI Commands
 
 Inspect queue:
 
@@ -163,7 +168,7 @@ python -m pybgworker.cli purge
 
 ---
 
-## 🧪 Observability
+## Observability
 
 PyBgWorker logs:
 
@@ -181,7 +186,7 @@ All machine-readable.
 
 ---
 
-## 🎯 Design Goals
+## Design Goals
 
 - zero external dependencies
 - SQLite durability
@@ -192,18 +197,23 @@ All machine-readable.
 
 ---
 
-## 📌 Roadmap
+## Roadmap
 
-Future upgrades may include:
+Planned but not yet included:
 
-- dashboard web UI
-- metrics endpoint
-- Redis backend
-- workflow pipelines
-- cluster coordination
+- Single-worker concurrency (process pool)
+- Retry backoff + jitter policies
+- Dead-letter queue for exhausted retries
+- Task/result TTL and automatic DB cleanup
+- Multiple named queues + routing
+- Pluggable backends (Redis first)
+- Cluster coordination / leader election for scheduler
+- Metrics endpoint and health checks
+- Dashboard API + web UI
+- Workflow pipelines / DAGs
 
 ---
 
-## 📄 License
+## License
 
 MIT License

{pybgworker-0.2.1 → pybgworker-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pybgworker"
-version = "0.2.1"
+version = "0.2.2"
 description = "Lightweight production-ready background task worker with cron, rate limiting and JSON observability"
 readme = "README.md"
 license = { text = "MIT" }

pybgworker-0.2.1/pybgworker/ratelimit.py

@@ -1,26 +0,0 @@
-import time
-import threading
-
-
-class RateLimiter:
-    def __init__(self, rate_per_sec):
-        self.rate = rate_per_sec
-        self.lock = threading.Lock()
-        self.timestamps = []
-
-    def acquire(self):
-        with self.lock:
-            now = time.time()
-
-            # remove old timestamps
-            self.timestamps = [
-                t for t in self.timestamps
-                if now - t < 1
-            ]
-
-            if len(self.timestamps) >= self.rate:
-                sleep_time = 1 - (now - self.timestamps[0])
-                if sleep_time > 0:
-                    time.sleep(sleep_time)
-
-            self.timestamps.append(time.time())

pybgworker-0.2.1/pybgworker/worker.py

@@ -1,122 +0,0 @@
-import time
-import traceback
-import threading
-from multiprocessing import Process, Queue as MPQueue
-
-from .logger import log
-from .sqlite_queue import SQLiteQueue
-from .task import TASK_REGISTRY
-from .config import WORKER_NAME, POLL_INTERVAL, RATE_LIMIT
-from .utils import loads, get_conn, now
-from .backends import SQLiteBackend
-from .scheduler import run_scheduler
-from .ratelimit import RateLimiter
-
-
-queue = SQLiteQueue()
-backend = SQLiteBackend()
-limiter = RateLimiter(RATE_LIMIT)
-
-TASK_TIMEOUT = 150  # seconds
-
-
-def heartbeat():
-    while True:
-        try:
-            with get_conn() as conn:
-                conn.execute("""
-                    INSERT INTO workers(name, last_seen)
-                    VALUES (?, ?)
-                    ON CONFLICT(name)
-                    DO UPDATE SET last_seen=excluded.last_seen
-                """, (WORKER_NAME, now().isoformat()))
-                conn.commit()
-        except Exception as e:
-            log("heartbeat_error", error=str(e))
-
-        time.sleep(5)
-
-
-def run_task(func, args, kwargs, result_queue):
-    try:
-        result = func(*args, **kwargs)
-        result_queue.put(("success", result))
-    except Exception:
-        result_queue.put(("error", traceback.format_exc()))
-
-
-def run_worker():
-    log("worker_start", worker=WORKER_NAME)
-
-    threading.Thread(target=heartbeat, daemon=True).start()
-    threading.Thread(target=run_scheduler, daemon=True).start()
-
-    while True:
-        task = queue.fetch_next(WORKER_NAME)
-
-        if not task:
-            time.sleep(POLL_INTERVAL)
-            continue
-
-        # ⭐ rate limiting happens here
-        limiter.acquire()
-
-        meta = TASK_REGISTRY.get(task["name"])
-        if not meta:
-            queue.fail(task["id"], "Task not registered")
-            log("task_invalid", task_id=task["id"])
-            continue
-
-        func = meta["func"]
-        retry_delay = meta["retry_delay"]
-
-        args = loads(task["args"])
-        kwargs = loads(task["kwargs"])
-
-        start_time = now()
-        log("task_start", task_id=task["id"], worker=WORKER_NAME)
-
-        result_queue = MPQueue()
-        process = Process(target=run_task, args=(func, args, kwargs, result_queue))
-
-        process.start()
-        process.join(TASK_TIMEOUT)
-
-        if process.is_alive():
-            process.terminate()
-
-            info = backend.get_task(task["id"])
-            if info["status"] == "cancelled":
-                log("task_cancelled", task_id=task["id"])
-                continue
-
-            queue.fail(task["id"], "Task timeout")
-            log("task_timeout", task_id=task["id"])
-            continue
-
-        if result_queue.empty():
-            queue.fail(task["id"], "Task crashed without result")
-            log("task_crash", task_id=task["id"])
-            continue
-
-        status, payload = result_queue.get()
-        duration = (now() - start_time).total_seconds()
-
-        if status == "success":
-            backend.store_result(task["id"], payload)
-            queue.ack(task["id"])
-            log("task_success",
-                task_id=task["id"],
-                duration=duration,
-                worker=WORKER_NAME)
-
-        else:
-            if task["attempt"] < task["max_retries"]:
-                queue.reschedule(task["id"], retry_delay)
-                log("task_retry",
-                    task_id=task["id"],
-                    attempt=task["attempt"] + 1,
-                    max=task["max_retries"])
-            else:
-                queue.fail(task["id"], payload)
-                log("task_failed", task_id=task["id"])