background-job-tracker 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Vishwajit Herma
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,330 @@
1
+ Metadata-Version: 2.4
2
+ Name: background-job-tracker
3
+ Version: 0.1.0
4
+ Summary: Official Python SDK for Background Job Tracker — real-time monitoring, telemetry, and incident detection for background tasks.
5
+ Author-email: Vishwajit Herma <vishuherma@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://background-job-tracker-wine.vercel.app/
8
+ Project-URL: Repository, https://github.com/Vishwajit-Herma/background-job-tracker
9
+ Keywords: background-jobs,celery,rq,redis-queue,monitoring,telemetry,observability,reliability,apm,incident-detection
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Intended Audience :: System Administrators
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.8
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Programming Language :: Python :: 3.14
22
+ Classifier: Framework :: Celery
23
+ Classifier: Topic :: System :: Monitoring
24
+ Classifier: Topic :: System :: Systems Administration
25
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
26
+ Classifier: Typing :: Typed
27
+ Requires-Python: >=3.8
28
+ Description-Content-Type: text/markdown
29
+ License-File: LICENSE
30
+ Requires-Dist: requests>=2.28.0
31
+ Provides-Extra: celery
32
+ Requires-Dist: celery>=5.0.0; extra == "celery"
33
+ Provides-Extra: rq
34
+ Requires-Dist: rq>=1.10.0; extra == "rq"
35
+ Provides-Extra: all
36
+ Requires-Dist: celery>=5.0.0; extra == "all"
37
+ Requires-Dist: rq>=1.10.0; extra == "all"
38
+ Provides-Extra: dev
39
+ Requires-Dist: pytest>=8.0.0; extra == "dev"
40
+ Requires-Dist: build>=1.0.0; extra == "dev"
41
+ Requires-Dist: twine>=5.0.0; extra == "dev"
42
+ Dynamic: license-file
43
+
44
+ # Background Job Tracker Python SDK
45
+
46
+ [![PyPI Version](https://img.shields.io/pypi/v/background-job-tracker.svg?color=blue)](https://pypi.org/project/background-job-tracker/)
47
+ [![Python Versions](https://img.shields.io/pypi/pyversions/background-job-tracker.svg)](https://pypi.org/project/background-job-tracker/)
48
+ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
49
+ [![Celery Support](https://img.shields.io/badge/Celery-5.0+-brightgreen.svg)](https://docs.celeryq.dev/)
50
+ [![RQ Support](https://img.shields.io/badge/RQ-1.10+-red.svg)](https://python-rq.org/)
51
+ [![Typing: Typed](https://img.shields.io/badge/Typing-PEP%20561-informational.svg)](https://peps.python.org/pep-0561/)
52
+
53
+ The official Python SDK for **Background Job Tracker (BJT)** — an end-to-end reliability, telemetry, and automated incident detection platform for background task workers.
54
+
55
+ Turn opaque background tasks into fully observable, auditable workflows. Automatically capture task durations, retries, failures, tracebacks, worker hostnames, and queue latencies without impacting application performance.
56
+
57
+ ---
58
+
59
+ ## Key Highlights
60
+
61
+ - ⚡ **Zero Performance Impact**: Telemetry is non-blocking. Events are placed in a memory-bounded local queue and flushed asynchronously by a background daemon thread.
62
+ - 🛡️ **Zero Failure Impact**: The SDK shields your tasks. Network partitions, timeouts, or 5xx backend errors never cause your customer-facing background tasks to fail.
63
+ - 🍴 **Process-Fork Safe**: Built-in PID change detection automatically resets isolation barriers when Celery forks worker processes (e.g. `prefork` pool), preventing shared lock or dead-thread issues.
64
+ - 🔄 **Intelligent Resilience**: Outgoing batches automatically retry with exponential backoff and jitter upon receiving `429 Too Many Requests` or transient `5xx` errors.
65
+ - 🔍 **Task Auto-Discovery**: Automatically syncs discovered task signatures and queues with your dashboard on worker startup.
66
+ - 📦 **First-Class Integrations**: Drop-in support for **Celery** and **Python RQ** with 3 lines of code.
67
+
68
+ ---
69
+
70
+ ## Installation
71
+
72
+ Install the base package with `pip`:
73
+
74
+ ```bash
75
+ pip install background-job-tracker
76
+ ```
77
+
78
+ ### With Framework Extras
79
+
80
+ Install with pre-configured dependencies for your task queue:
81
+
82
+ ```bash
83
+ # For Celery
84
+ pip install "background-job-tracker[celery]"
85
+
86
+ # For Python RQ (Redis Queue)
87
+ pip install "background-job-tracker[rq]"
88
+
89
+ # For all supported frameworks
90
+ pip install "background-job-tracker[all]"
91
+ ```
92
+
93
+ Using `uv`:
94
+ ```bash
95
+ uv add background-job-tracker
96
+ ```
97
+
98
+ Using `poetry`:
99
+ ```bash
100
+ poetry add background-job-tracker
101
+ ```
102
+
103
+ ---
104
+
105
+ ## Quickstart
106
+
107
+ ### 1. Celery (Django or Standalone)
108
+
109
+ Integrate Background Job Tracker into your Celery application in 3 lines of code.
110
+
111
+ #### Step 1: Set Your Environment Variables
112
+ Export your project credentials (found in your Background Job Tracker dashboard):
113
+
114
+ ```bash
115
+ export BACKGROUND_JOB_TRACKER_API_KEY="bjt_live_xxxxxxxxxxxxxxxx"
116
+ export BACKGROUND_JOB_TRACKER_BASE_URL="https://app.jobtracker.io" # or http://localhost:8000 for self-hosted
117
+ ```
118
+
119
+ #### Step 2: Initialize in `celery.py`
120
+
121
+ In your Django `celery.py` (or wherever your `Celery` app instance is configured):
122
+
123
+ ```python
124
+ import os
125
+ from celery import Celery
126
+ from background_job_tracker import Tracker
127
+ from background_job_tracker.integrations.celery import CeleryIntegration
128
+
129
+ # Standard Celery application setup
130
+ app = Celery("my_project")
131
+ app.config_from_object("django.conf:settings", namespace="CELERY")
132
+ app.autodiscover_tasks()
133
+
134
+ # Initialize the Background Job Tracker telemetry
135
+ tracker = Tracker() # Automatically reads BACKGROUND_JOB_TRACKER_API_KEY from environment
136
+ CeleryIntegration(app=app, tracker=tracker)
137
+ ```
138
+
139
+ That's it! When your Celery worker starts up:
140
+ 1. `worker_ready` signal auto-registers all defined Celery tasks with the platform.
141
+ 2. `task_prerun`, `task_postrun`, `task_success`, `task_failure`, `task_retry`, and `task_revoked` signals automatically report lifecycle executions.
142
+
143
+ ---
144
+
145
+ ### 2. Python RQ (Redis Queue)
146
+
147
+ To monitor Python RQ workers, attach the `RQIntegration` to your RQ worker script:
148
+
149
+ ```python
150
+ import os
151
+ from redis import Redis
152
+ from rq import Worker, Queue
153
+ from background_job_tracker import Tracker
154
+ from background_job_tracker.integrations.rq import RQIntegration
155
+
156
+ # 1. Initialize Tracker
157
+ tracker = Tracker(
158
+ api_key=os.environ["BACKGROUND_JOB_TRACKER_API_KEY"],
159
+ base_url=os.getenv("BACKGROUND_JOB_TRACKER_BASE_URL", "https://app.jobtracker.io"),
160
+ )
161
+
162
+ # 2. Configure RQ Integration
163
+ # Pass modules or functions to automatically discover task names
164
+ rq_integration = RQIntegration(
165
+ tracker=tracker,
166
+ modules=["my_app.tasks"],
167
+ )
168
+
169
+ # 3. Attach integration to your worker
170
+ if __name__ == "__main__":
171
+ redis_conn = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379/0"))
172
+ queue = Queue("default", connection=redis_conn)
173
+
174
+ worker = Worker([queue], connection=redis_conn)
175
+ rq_integration.attach(worker)
176
+
177
+ # Start processing jobs
178
+ worker.work(with_scheduler=True)
179
+ ```
180
+
181
+ Alternatively, use the convenient one-liner helper `setup_rq`:
182
+
183
+ ```python
184
+ from background_job_tracker.integrations.rq import setup_rq
185
+
186
+ setup_rq(worker, tracker=tracker, modules=["my_app.tasks"])
187
+ worker.work()
188
+ ```
189
+
190
+ ---
191
+
192
+ ### 3. Manual / Custom Task Tracking
193
+
194
+ If you run custom background threads, asyncio workers, or non-standard queue systems, you can directly enqueue execution events:
195
+
196
+ ```python
197
+ from background_job_tracker import Tracker
198
+
199
+ tracker = Tracker(api_key="bjt_live_xxxxxxxxxxxxxxxx")
200
+
201
+ # Track an execution event
202
+ tracker.enqueue_event(
203
+ {
204
+ "task_identifier": "reports.generate_monthly_pdf",
205
+ "external_id": "job_984572049",
206
+ "status": "success", # "running" | "success" | "failure" | "retry" | "revoked"
207
+ "duration_seconds": 2.45,
208
+ "started_at": "2026-09-10T12:00:00.000000Z",
209
+ "completed_at": "2026-09-10T12:00:02.450000Z",
210
+ "worker": "worker-node-1",
211
+ "queue": "reports",
212
+ "retry_count": 0,
213
+ "metadata": {
214
+ "report_id": 402,
215
+ "format": "pdf",
216
+ },
217
+ }
218
+ )
219
+
220
+ # In CLI scripts or shutdown hooks, ensure all events are flushed:
221
+ tracker.shutdown(timeout=5.0)
222
+ ```
223
+
224
+ ---
225
+
226
+ ## Configuration Reference
227
+
228
+ The `Tracker` client can be configured programmatically or via environment variables:
229
+
230
+ | Setting | Environment Variable | Default | Description |
231
+ |---|---|---|---|
232
+ | **API Key** | `BACKGROUND_JOB_TRACKER_API_KEY` (or `BJT_SDK_API_KEY`) | *Required* | Project API key generated in the BJT Dashboard. |
233
+ | **Base URL** | `BACKGROUND_JOB_TRACKER_BASE_URL` | `http://localhost:8000` | SaaS ingestion endpoint or self-hosted server URL. |
234
+ | **Batch Size** | `batch_size` | `100` | Maximum number of events bundled into a single HTTP POST request. |
235
+ | **Flush Interval** | `flush_interval` | `5.0` | Maximum seconds to wait before flushing an incomplete batch. |
236
+ | **Max Queue Size** | `max_queue_size` | `10000` | In-memory queue limit. If reached, new events are safely dropped. |
237
+ | **Discovery Interval**| `task_discovery_interval` | `300.0` | Seconds between periodic background task discovery syncs. |
238
+
239
+ ### Example with Explicit Options
240
+
241
+ ```python
242
+ tracker = Tracker(
243
+ api_key="bjt_live_xxxxxxxxxxxxxxxx",
244
+ base_url="https://app.jobtracker.io",
245
+ batch_size=50, # Flush after 50 events
246
+ flush_interval=2.0, # Or flush every 2 seconds
247
+ max_queue_size=20000, # Buffer up to 20,000 events in memory
248
+ task_discovery_interval=600.0, # Sync task list every 10 minutes
249
+ )
250
+ ```
251
+
252
+ ---
253
+
254
+ ## Architecture & Zero-Impact Guarantee
255
+
256
+ ```
257
+ ┌─────────────────────────────────────────────────────────────────┐
258
+ │ Customer Worker Process │
259
+ │ │
260
+ │ [Celery / RQ Task] │
261
+ │ │ │
262
+ │ ▼ (Signals / Hooks) │
263
+ │ [Tracker.enqueue_event] │
264
+ │ │ (Non-blocking put_nowait < 0.01ms) │
265
+ │ ▼ │
266
+ │ ┌───────────────┐ │
267
+ │ │ Bounded Queue │ (If queue is full: safely drops event, │
268
+ │ └───────┬───────┘ never blocks execution or starves memory) │
269
+ │ │ │
270
+ │ │ (Internal thread consumption) │
271
+ │ ▼ │
272
+ │ ┌─────────────────────────────┐ │
273
+ │ │ BackgroundSender Thread │ │
274
+ │ │ - Batches events │ │
275
+ │ │ - Handles HTTP retry logic │ │
276
+ │ │ - Exponential backoff │ │
277
+ │ └─────────────┬───────────────┘ │
278
+ └─────────────────┼───────────────────────────────────────────────┘
279
+
280
+ ▼ (Asynchronous HTTPS POST)
281
+ ┌────────────────────────┐
282
+ │ Background Job Tracker │
283
+ │ SaaS Backend │
284
+ └────────────────────────┘
285
+ ```
286
+
287
+ 1. **Non-blocking queue enqueue**: When a task completes, the signal handler performs a `put_nowait()` on an in-memory `queue.Queue`. This completes in microseconds (< 0.01ms).
288
+ 2. **Dedicated daemon sender thread**: A separate daemon thread extracts batches and transmits them via HTTP.
289
+ 3. **Queue overflow drop protection**: If the network connection goes down and the queue reaches `max_queue_size` (10,000 items), subsequent events are dropped with a warning. Your workers will **never** run out of memory or pause task processing.
290
+ 4. **Fork safety in Celery**: Celery prefork workers use `fork()` without `exec()`. Background threads do not survive forks. The SDK detects the PID change upon the first signal inside a child worker and automatically spawns a dedicated queue and sender thread for that process.
291
+
292
+ ---
293
+
294
+ ## Debugging & Logging
295
+
296
+ The SDK logs diagnostic messages through the standard Python `logging` module under the logger hierarchy `background_job_tracker`:
297
+
298
+ ```python
299
+ import logging
300
+
301
+ # Enable verbose logging for the SDK
302
+ logging.getLogger("background_job_tracker").setLevel(logging.DEBUG)
303
+ handler = logging.StreamHandler()
304
+ handler.setFormatter(logging.Formatter("[%(asctime)s] [%(name)s] [%(levelname)s] %(message)s"))
305
+ logging.getLogger("background_job_tracker").addHandler(handler)
306
+ ```
307
+
308
+ ---
309
+
310
+ ## Testing & Verification
311
+
312
+ To run unit tests locally:
313
+
314
+ ```bash
315
+ # Clone the repository
316
+ git clone https://github.com/Vishwajit-Herma/background-job-tracker.git
317
+ cd background-job-tracker/sdk
318
+
319
+ # Install dependencies and test runner
320
+ pip install -e ".[all,dev]"
321
+
322
+ # Run tests
323
+ pytest
324
+ ```
325
+
326
+ ---
327
+
328
+ ## License
329
+
330
+ This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
@@ -0,0 +1,287 @@
1
+ # Background Job Tracker Python SDK
2
+
3
+ [![PyPI Version](https://img.shields.io/pypi/v/background-job-tracker.svg?color=blue)](https://pypi.org/project/background-job-tracker/)
4
+ [![Python Versions](https://img.shields.io/pypi/pyversions/background-job-tracker.svg)](https://pypi.org/project/background-job-tracker/)
5
+ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
6
+ [![Celery Support](https://img.shields.io/badge/Celery-5.0+-brightgreen.svg)](https://docs.celeryq.dev/)
7
+ [![RQ Support](https://img.shields.io/badge/RQ-1.10+-red.svg)](https://python-rq.org/)
8
+ [![Typing: Typed](https://img.shields.io/badge/Typing-PEP%20561-informational.svg)](https://peps.python.org/pep-0561/)
9
+
10
+ The official Python SDK for **Background Job Tracker (BJT)** — an end-to-end reliability, telemetry, and automated incident detection platform for background task workers.
11
+
12
+ Turn opaque background tasks into fully observable, auditable workflows. Automatically capture task durations, retries, failures, tracebacks, worker hostnames, and queue latencies without impacting application performance.
13
+
14
+ ---
15
+
16
+ ## Key Highlights
17
+
18
+ - ⚡ **Zero Performance Impact**: Telemetry is non-blocking. Events are placed in a memory-bounded local queue and flushed asynchronously by a background daemon thread.
19
+ - 🛡️ **Zero Failure Impact**: The SDK shields your tasks. Network partitions, timeouts, or 5xx backend errors never cause your customer-facing background tasks to fail.
20
+ - 🍴 **Process-Fork Safe**: Built-in PID change detection automatically resets isolation barriers when Celery forks worker processes (e.g. `prefork` pool), preventing shared lock or dead-thread issues.
21
+ - 🔄 **Intelligent Resilience**: Outgoing batches automatically retry with exponential backoff and jitter upon receiving `429 Too Many Requests` or transient `5xx` errors.
22
+ - 🔍 **Task Auto-Discovery**: Automatically syncs discovered task signatures and queues with your dashboard on worker startup.
23
+ - 📦 **First-Class Integrations**: Drop-in support for **Celery** and **Python RQ** with 3 lines of code.
24
+
25
+ ---
26
+
27
+ ## Installation
28
+
29
+ Install the base package with `pip`:
30
+
31
+ ```bash
32
+ pip install background-job-tracker
33
+ ```
34
+
35
+ ### With Framework Extras
36
+
37
+ Install with pre-configured dependencies for your task queue:
38
+
39
+ ```bash
40
+ # For Celery
41
+ pip install "background-job-tracker[celery]"
42
+
43
+ # For Python RQ (Redis Queue)
44
+ pip install "background-job-tracker[rq]"
45
+
46
+ # For all supported frameworks
47
+ pip install "background-job-tracker[all]"
48
+ ```
49
+
50
+ Using `uv`:
51
+ ```bash
52
+ uv add background-job-tracker
53
+ ```
54
+
55
+ Using `poetry`:
56
+ ```bash
57
+ poetry add background-job-tracker
58
+ ```
59
+
60
+ ---
61
+
62
+ ## Quickstart
63
+
64
+ ### 1. Celery (Django or Standalone)
65
+
66
+ Integrate Background Job Tracker into your Celery application in 3 lines of code.
67
+
68
+ #### Step 1: Set Your Environment Variables
69
+ Export your project credentials (found in your Background Job Tracker dashboard):
70
+
71
+ ```bash
72
+ export BACKGROUND_JOB_TRACKER_API_KEY="bjt_live_xxxxxxxxxxxxxxxx"
73
+ export BACKGROUND_JOB_TRACKER_BASE_URL="https://app.jobtracker.io" # or http://localhost:8000 for self-hosted
74
+ ```
75
+
76
+ #### Step 2: Initialize in `celery.py`
77
+
78
+ In your Django `celery.py` (or wherever your `Celery` app instance is configured):
79
+
80
+ ```python
81
+ import os
82
+ from celery import Celery
83
+ from background_job_tracker import Tracker
84
+ from background_job_tracker.integrations.celery import CeleryIntegration
85
+
86
+ # Standard Celery application setup
87
+ app = Celery("my_project")
88
+ app.config_from_object("django.conf:settings", namespace="CELERY")
89
+ app.autodiscover_tasks()
90
+
91
+ # Initialize the Background Job Tracker telemetry
92
+ tracker = Tracker() # Automatically reads BACKGROUND_JOB_TRACKER_API_KEY from environment
93
+ CeleryIntegration(app=app, tracker=tracker)
94
+ ```
95
+
96
+ That's it! When your Celery worker starts up:
97
+ 1. `worker_ready` signal auto-registers all defined Celery tasks with the platform.
98
+ 2. `task_prerun`, `task_postrun`, `task_success`, `task_failure`, `task_retry`, and `task_revoked` signals automatically report lifecycle executions.
99
+
100
+ ---
101
+
102
+ ### 2. Python RQ (Redis Queue)
103
+
104
+ To monitor Python RQ workers, attach the `RQIntegration` to your RQ worker script:
105
+
106
+ ```python
107
+ import os
108
+ from redis import Redis
109
+ from rq import Worker, Queue
110
+ from background_job_tracker import Tracker
111
+ from background_job_tracker.integrations.rq import RQIntegration
112
+
113
+ # 1. Initialize Tracker
114
+ tracker = Tracker(
115
+ api_key=os.environ["BACKGROUND_JOB_TRACKER_API_KEY"],
116
+ base_url=os.getenv("BACKGROUND_JOB_TRACKER_BASE_URL", "https://app.jobtracker.io"),
117
+ )
118
+
119
+ # 2. Configure RQ Integration
120
+ # Pass modules or functions to automatically discover task names
121
+ rq_integration = RQIntegration(
122
+ tracker=tracker,
123
+ modules=["my_app.tasks"],
124
+ )
125
+
126
+ # 3. Attach integration to your worker
127
+ if __name__ == "__main__":
128
+ redis_conn = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379/0"))
129
+ queue = Queue("default", connection=redis_conn)
130
+
131
+ worker = Worker([queue], connection=redis_conn)
132
+ rq_integration.attach(worker)
133
+
134
+ # Start processing jobs
135
+ worker.work(with_scheduler=True)
136
+ ```
137
+
138
+ Alternatively, use the convenient one-liner helper `setup_rq`:
139
+
140
+ ```python
141
+ from background_job_tracker.integrations.rq import setup_rq
142
+
143
+ setup_rq(worker, tracker=tracker, modules=["my_app.tasks"])
144
+ worker.work()
145
+ ```
146
+
147
+ ---
148
+
149
+ ### 3. Manual / Custom Task Tracking
150
+
151
+ If you run custom background threads, asyncio workers, or non-standard queue systems, you can directly enqueue execution events:
152
+
153
+ ```python
154
+ from background_job_tracker import Tracker
155
+
156
+ tracker = Tracker(api_key="bjt_live_xxxxxxxxxxxxxxxx")
157
+
158
+ # Track an execution event
159
+ tracker.enqueue_event(
160
+ {
161
+ "task_identifier": "reports.generate_monthly_pdf",
162
+ "external_id": "job_984572049",
163
+ "status": "success", # "running" | "success" | "failure" | "retry" | "revoked"
164
+ "duration_seconds": 2.45,
165
+ "started_at": "2026-09-10T12:00:00.000000Z",
166
+ "completed_at": "2026-09-10T12:00:02.450000Z",
167
+ "worker": "worker-node-1",
168
+ "queue": "reports",
169
+ "retry_count": 0,
170
+ "metadata": {
171
+ "report_id": 402,
172
+ "format": "pdf",
173
+ },
174
+ }
175
+ )
176
+
177
+ # In CLI scripts or shutdown hooks, ensure all events are flushed:
178
+ tracker.shutdown(timeout=5.0)
179
+ ```
180
+
181
+ ---
182
+
183
+ ## Configuration Reference
184
+
185
+ The `Tracker` client can be configured programmatically or via environment variables:
186
+
187
+ | Setting | Environment Variable | Default | Description |
188
+ |---|---|---|---|
189
+ | **API Key** | `BACKGROUND_JOB_TRACKER_API_KEY` (or `BJT_SDK_API_KEY`) | *Required* | Project API key generated in the BJT Dashboard. |
190
+ | **Base URL** | `BACKGROUND_JOB_TRACKER_BASE_URL` | `http://localhost:8000` | SaaS ingestion endpoint or self-hosted server URL. |
191
+ | **Batch Size** | `batch_size` | `100` | Maximum number of events bundled into a single HTTP POST request. |
192
+ | **Flush Interval** | `flush_interval` | `5.0` | Maximum seconds to wait before flushing an incomplete batch. |
193
+ | **Max Queue Size** | `max_queue_size` | `10000` | In-memory queue limit. If reached, new events are safely dropped. |
194
+ | **Discovery Interval**| `task_discovery_interval` | `300.0` | Seconds between periodic background task discovery syncs. |
195
+
196
+ ### Example with Explicit Options
197
+
198
+ ```python
199
+ tracker = Tracker(
200
+ api_key="bjt_live_xxxxxxxxxxxxxxxx",
201
+ base_url="https://app.jobtracker.io",
202
+ batch_size=50, # Flush after 50 events
203
+ flush_interval=2.0, # Or flush every 2 seconds
204
+ max_queue_size=20000, # Buffer up to 20,000 events in memory
205
+ task_discovery_interval=600.0, # Sync task list every 10 minutes
206
+ )
207
+ ```
208
+
209
+ ---
210
+
211
+ ## Architecture & Zero-Impact Guarantee
212
+
213
+ ```
214
+ ┌─────────────────────────────────────────────────────────────────┐
215
+ │ Customer Worker Process │
216
+ │ │
217
+ │ [Celery / RQ Task] │
218
+ │ │ │
219
+ │ ▼ (Signals / Hooks) │
220
+ │ [Tracker.enqueue_event] │
221
+ │ │ (Non-blocking put_nowait < 0.01ms) │
222
+ │ ▼ │
223
+ │ ┌───────────────┐ │
224
+ │ │ Bounded Queue │ (If queue is full: safely drops event, │
225
+ │ └───────┬───────┘ never blocks execution or starves memory) │
226
+ │ │ │
227
+ │ │ (Internal thread consumption) │
228
+ │ ▼ │
229
+ │ ┌─────────────────────────────┐ │
230
+ │ │ BackgroundSender Thread │ │
231
+ │ │ - Batches events │ │
232
+ │ │ - Handles HTTP retry logic │ │
233
+ │ │ - Exponential backoff │ │
234
+ │ └─────────────┬───────────────┘ │
235
+ └─────────────────┼───────────────────────────────────────────────┘
236
+
237
+ ▼ (Asynchronous HTTPS POST)
238
+ ┌────────────────────────┐
239
+ │ Background Job Tracker │
240
+ │ SaaS Backend │
241
+ └────────────────────────┘
242
+ ```
243
+
244
+ 1. **Non-blocking queue enqueue**: When a task completes, the signal handler performs a `put_nowait()` on an in-memory `queue.Queue`. This completes in microseconds (< 0.01ms).
245
+ 2. **Dedicated daemon sender thread**: A separate daemon thread extracts batches and transmits them via HTTP.
246
+ 3. **Queue overflow drop protection**: If the network connection goes down and the queue reaches `max_queue_size` (10,000 items), subsequent events are dropped with a warning. Your workers will **never** run out of memory or pause task processing.
247
+ 4. **Fork safety in Celery**: Celery prefork workers use `fork()` without `exec()`. Background threads do not survive forks. The SDK detects the PID change upon the first signal inside a child worker and automatically spawns a dedicated queue and sender thread for that process.
248
+
249
+ ---
250
+
251
+ ## Debugging & Logging
252
+
253
+ The SDK logs diagnostic messages through the standard Python `logging` module under the logger hierarchy `background_job_tracker`:
254
+
255
+ ```python
256
+ import logging
257
+
258
+ # Enable verbose logging for the SDK
259
+ logging.getLogger("background_job_tracker").setLevel(logging.DEBUG)
260
+ handler = logging.StreamHandler()
261
+ handler.setFormatter(logging.Formatter("[%(asctime)s] [%(name)s] [%(levelname)s] %(message)s"))
262
+ logging.getLogger("background_job_tracker").addHandler(handler)
263
+ ```
264
+
265
+ ---
266
+
267
+ ## Testing & Verification
268
+
269
+ To run unit tests locally:
270
+
271
+ ```bash
272
+ # Clone the repository
273
+ git clone https://github.com/Vishwajit-Herma/background-job-tracker.git
274
+ cd background-job-tracker/sdk
275
+
276
+ # Install dependencies and test runner
277
+ pip install -e ".[all,dev]"
278
+
279
+ # Run tests
280
+ pytest
281
+ ```
282
+
283
+ ---
284
+
285
+ ## License
286
+
287
+ This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
@@ -0,0 +1,11 @@
1
+ """
2
+ Background Job Tracker Python SDK.
3
+
4
+ Real-time reliability, observability, and incident detection for Celery,
5
+ RQ, and background task queues.
6
+ """
7
+
8
+ from .client import Tracker
9
+
10
+ __version__ = "0.1.0"
11
+ __all__ = ["Tracker", "__version__"]