queuelens 0.1.0__tar.gz

queuelens-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,73 @@
+
+name: Publish QLens SDK to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+  # Allows us to test the build without publishing.
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    name: Build SDK distributions
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+
+      - name: Install build dependencies
+        run: |
+          python -m pip install --upgrade build twine
+
+      - name: Build distributions
+        run: |
+          python -m build
+
+      - name: Validate distributions
+        run: |
+          python -m twine check dist/*
+
+      - name: Upload distributions
+        uses: actions/upload-artifact@v6
+        with:
+          name: python-package-distributions
+          path: dist/
+          if-no-files-found: error
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+
+    # Only publish when a version tag is pushed.
+    if: startsWith(github.ref, 'refs/tags/v')
+
+    environment:
+      name: pypi
+      url: https://pypi.org/p/queuelens
+
+    permissions:
+      id-token: write
+
+    steps:
+      - name: Download distributions
+        uses: actions/download-artifact@v6
+        with:
+          name: python-package-distributions
+          path: dist/
+
+      - name: Publish package to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

queuelens-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+# Virtual environments
+.venv/
+.venv-*/
+venv/
+
+# Build artifacts
+dist/
+build/
+*.egg-info/
+
+# Python cache
+__pycache__/
+*.py[cod]
+
+# Test and tooling caches
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/

queuelens-0.1.0/DELIVERY.md

@@ -0,0 +1,187 @@
+
+# QueueLens SDK — Event Delivery Guarantees
+
+## Overview
+
+The QueueLens SDK captures Celery task lifecycle events and asynchronously
+transmits them to the QueueLens ingestion API.
+
+The SDK is designed to observe customer workloads without becoming a
+dependency of task execution.
+
+Event delivery uses an in-memory bounded buffer, background HTTP delivery,
+and retry handling.
+
+## 1. Event buffering
+
+The SDK maintains an in-memory event buffer.
+
+Default configuration:
+
+| Setting | Default |
+|---|---:|
+| Buffer capacity | 1,000 events |
+| Batch size | 100 events |
+| Flush interval | 5 seconds |
+| Maximum batches per cycle | 10 |
+
+Events are added to the buffer when Celery lifecycle signals are received.
+
+The buffer is bounded to prevent unbounded memory consumption.
+
+### Buffer overflow
+
+When the buffer reaches capacity:
+
+- New events are rejected.
+- Existing buffered events are retained.
+- The dropped event counter is incremented.
+- A warning is logged.
+
+The SDK exposes the dropped event count through its instrumentation instance.
+
+## 2. Event transmission
+
+The SDK sends buffered events to the QueueLens ingestion API using HTTP.
+
+Each request contains a batch of events.
+
+Authentication uses the configured QueueLens API key.
+
+The SDK acknowledges events only after:
+
+1. The HTTP request succeeds.
+2. The ingestion response can be validated.
+3. The accepted and duplicate counts match the submitted batch size.
+
+Events that fail delivery remain in the buffer.
+
+## 3. Retry behavior
+
+When event delivery fails, the SDK retains the events for another attempt.
+
+The background flusher uses exponential backoff with equal jitter.
+
+Default retry settings:
+
+| Setting | Default |
+|---|---:|
+| Initial backoff | 1 second |
+| Maximum backoff | 30 seconds |
+| Retry strategy | Exponential backoff with equal jitter |
+
+A successful delivery cycle resets the retry backoff.
+
+Retry attempts continue while the worker process remains alive.
+
+## 4. Duplicate event handling
+
+Each event has a unique event ID.
+
+The ingestion backend uses this ID to identify duplicate events.
+
+If an event was committed by the backend but the SDK did not receive the response, the SDK may transmit that event again.
+
+The backend deduplicates the repeated event ID.
+
+This provides duplicate-safe ingestion, but does not guarantee exactly-once delivery.
+
+## 5. Shutdown behavior
+
+During graceful shutdown:
+
+1. The SDK signals the background flusher to stop.
+2. The flusher waits for an in-progress delivery operation to finish.
+3. The flusher makes one final delivery attempt.
+4. The HTTP client is closed after the flusher exits.
+
+The shutdown operation has a configurable timeout.
+
+If the flusher does not stop within that timeout, it may continue running in the background until its current operation completes.
+
+A shutdown timeout does not forcibly terminate the background thread.
+
+## 6. Process termination and data loss
+
+The current SDK uses in-memory buffering.
+
+Pending events may be lost when:
+
+- A worker process crashes.
+- A container is forcefully terminated.
+- The operating system kills the process.
+- A machine loses power.
+- The process exits before buffered events are delivered.
+
+The SDK does not currently persist pending events to disk.
+
+Graceful shutdown attempts to deliver pending events, but a final failed attempt can still leave events undelivered.
+
+## 7. Failure isolation
+
+QueueLens instrumentation failures must not interrupt customer task execution.
+
+The SDK handles its own event capture and delivery failures independently.
+
+Examples include:
+
+- Event serialization failures.
+- Buffer insertion failures.
+- HTTP transport failures.
+- Invalid ingestion responses.
+- Background flusher initialization failures.
+- Shutdown failures.
+
+QueueLens does not suppress exceptions raised by the customer's actual Celery task.
+
+## 8. Delivery guarantee
+
+The current SDK provides:
+
+- In-memory event buffering.
+- Bounded memory usage.
+- Background HTTP transmission.
+- Retention and retries during process lifetime.
+- Duplicate-safe ingestion.
+- Best-effort graceful shutdown delivery.
+
+The current SDK does not provide:
+
+- Durable event storage.
+- Guaranteed delivery after process crashes.
+- Exactly-once delivery.
+- Guaranteed delivery when the backend remains unavailable.
+- Persistent recovery of buffered events after a restart.
+
+## 9. Configuration example
+
+The SDK can be configured through environment variables.
+
+```bash
+QUEUELENS_API_KEY=ql_test_your_api_key
+QUEUELENS_ENDPOINT=https://api.example.com/internal/v1/events/batch
+
+QUEUELENS_BATCH_SIZE=100
+QUEUELENS_FLUSH_INTERVAL=5
+QUEUELENS_TIMEOUT=5
+QUEUELENS_BUFFER_CAPACITY=1000
+QUEUELENS_MAX_BATCHES_PER_CYCLE=10
+QUEUELENS_SHUTDOWN_TIMEOUT=10
+```
+
+Explicit constructor arguments take precedence over environment variables.
+
+See the SDK configuration reference for all supported settings.
+
+## 10. Future improvements
+
+Potential future reliability improvements include:
+
+- Optional disk-backed event persistence.
+- Recovery of pending events after process restart.
+- Configurable overflow strategies.
+- Delivery health metrics.
+- Persistent delivery acknowledgments.
+- Improved visibility into dropped and undelivered events.
+
+These features are not part of the current SDK implementation.

queuelens-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 [Your Legal Name or Company Name]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

queuelens-0.1.0/PKG-INFO

@@ -0,0 +1,258 @@
+Metadata-Version: 2.5
+Name: queuelens
+Version: 0.1.0
+Summary: Python SDK for background job observability and Celery instrumentation
+Project-URL: Homepage, https://github.com/muhammedsafuvan/qlens-sdk
+Project-URL: Repository, https://github.com/muhammedsafuvan/qlens-sdk
+Project-URL: Issues, https://github.com/muhammedsafuvan/qlens-sdk/issues
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: pydantic>=2.0
+Provides-Extra: celery
+Requires-Dist: celery[redis]>=5.6.3; extra == 'celery'
+Description-Content-Type: text/markdown
+
+# QueueLens Python SDK
+
+QueueLens provides background job observability for Python applications using Celery.
+
+It instruments Celery task lifecycle events and asynchronously delivers them to your QueueLens instance, allowing you to monitor job execution, failures, retries, and worker activity.
+
+## Requirements
+
+- Python 3.12+
+- Celery 5.6+
+- A running QueueLens instance
+- A QueueLens workspace API key
+
+## Installation
+
+Install the SDK with Celery support:
+
+```bash
+pip install "queuelens[celery]"
+```
+
+This installs QueueLens and its optional Celery integration dependencies.
+
+## Quick start
+
+The recommended integration is **Standard Integration**.
+
+### 1. Configure environment variables
+
+```bash
+export QUEUELENS_API_KEY="ql_test_your_api_key"
+export QUEUELENS_ENDPOINT="http://localhost:8001/internal/v1/events/batch"
+```
+
+Replace the example API key and endpoint with the credentials and ingestion endpoint provided by your QueueLens deployment.
+
+### 2. Instrument your Celery application
+
+```python
+from celery import Celery
+from queuelens import QueueLens
+
+app = Celery("orders")
+
+queuelens = QueueLens()
+queuelens.instrument(app)
+```
+
+That's it. QueueLens automatically registers its Celery signal handlers and manages event delivery through a background flusher.
+
+You do not need to modify your existing task definitions.
+
+### 3. Run your worker
+
+Start your Celery worker as usual:
+
+```bash
+celery -A your_module.app worker --loglevel=info
+```
+
+Your task lifecycle events will be captured and delivered to QueueLens.
+
+## Configuration
+
+QueueLens supports configuration through constructor arguments or environment variables.
+
+Constructor arguments take precedence over environment variables, which take precedence over defaults.
+
+| Parameter | Environment variable | Default |
+|---|---|---:|
+| `api_key` | `QUEUELENS_API_KEY` | Required |
+| `endpoint` | `QUEUELENS_ENDPOINT` | Required |
+| `batch_size` | `QUEUELENS_BATCH_SIZE` | 100 |
+| `flush_interval` | `QUEUELENS_FLUSH_INTERVAL` | 5 seconds |
+| `timeout` | `QUEUELENS_TIMEOUT` | 5 seconds |
+| `buffer_capacity` | `QUEUELENS_BUFFER_CAPACITY` | 1,000 |
+| `max_batches_per_cycle` | `QUEUELENS_MAX_BATCHES_PER_CYCLE` | 10 |
+| `shutdown_timeout` | `QUEUELENS_SHUTDOWN_TIMEOUT` | 10 seconds |
+
+### Configure using Python
+
+```python
+from celery import Celery
+from queuelens import QueueLens
+
+app = Celery("orders")
+
+queuelens = QueueLens(
+    api_key="ql_test_your_api_key",
+    endpoint="http://localhost:8001/internal/v1/events/batch",
+    batch_size=100,
+    flush_interval=5,
+    timeout=5,
+)
+
+queuelens.instrument(app)
+```
+
+### Configure using environment variables
+
+```bash
+export QUEUELENS_API_KEY="ql_test_your_api_key"
+export QUEUELENS_ENDPOINT="http://localhost:8001/internal/v1/events/batch"
+
+export QUEUELENS_BATCH_SIZE=100
+export QUEUELENS_FLUSH_INTERVAL=5
+export QUEUELENS_TIMEOUT=5
+export QUEUELENS_BUFFER_CAPACITY=1000
+export QUEUELENS_MAX_BATCHES_PER_CYCLE=10
+export QUEUELENS_SHUTDOWN_TIMEOUT=10
+```
+
+Keep API keys in environment variables or a secrets manager. Do not commit credentials to source control.
+
+## Captured events
+
+QueueLens captures the following Celery task lifecycle events:
+
+| Event | Description |
+|---|---|
+| `received` | Task received by a worker |
+| `started` | Task execution started |
+| `succeeded` | Task completed successfully |
+| `failed` | Task execution failed |
+| `retried` | Task was scheduled for retry |
+
+Each event includes available metadata such as:
+
+- Task ID
+- Task name
+- Event type
+- Event timestamp
+- Worker name
+- Queue name
+- Attempt number
+
+The attempt number is derived from Celery's retry count.
+
+## Event delivery
+
+QueueLens delivers events asynchronously so that instrumentation does not need to wait for HTTP requests during task execution.
+
+The SDK uses:
+
+- A bounded in-memory event buffer.
+- Configurable batch sizes.
+- Periodic background flushing.
+- HTTP timeouts.
+- Automatic retries with exponential backoff and jitter.
+- Event acknowledgement after successful ingestion.
+- A final delivery attempt during graceful shutdown.
+
+### Retry behavior
+
+When delivery fails, events remain in the buffer and are retried.
+
+The retry delay uses exponential backoff, starting at approximately one second and increasing up to 30 seconds, with jitter.
+
+Events are acknowledged only after the ingestion response is successfully received and validated.
+
+QueueLens uses event IDs to support duplicate-safe ingestion. This provides retry-safe delivery behavior, not exactly-once delivery.
+
+## Graceful shutdown
+
+QueueLens automatically integrates with Celery worker lifecycle signals.
+
+When a worker shuts down, the SDK:
+
+1. Stops the background event flusher.
+2. Attempts to deliver remaining buffered events.
+3. Closes its HTTP client.
+
+You generally do not need to manage the SDK lifecycle manually.
+
+For explicit cleanup, you can call:
+
+```python
+queuelens.close()
+```
+
+Do not call this while the Celery worker is still expected to capture events.
+
+## Reliability and limitations
+
+QueueLens is designed to keep observability failures isolated from application task execution.
+
+However, there are important limitations:
+
+- **In-memory buffering:** Events are held in process memory and are not persisted to disk.
+- **Process termination:** A crash, forceful termination, or machine failure can result in loss of pending events.
+- **Buffer capacity:** When the buffer is full, new events are dropped and the SDK logs a warning.
+- **Network failures:** Failed deliveries are retried while the process remains alive and events remain buffered.
+- **Graceful shutdown:** The SDK makes a final delivery attempt, but delivery is not guaranteed if the endpoint remains unavailable or the shutdown timeout is exceeded.
+
+The SDK should be treated as an observability integration, not as a replacement for Celery's task reliability mechanisms.
+
+## Logging
+
+QueueLens uses Python's standard logging module.
+
+To enable debug-level SDK logs:
+
+```python
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logging.getLogger("queuelens").setLevel(logging.DEBUG)
+```
+
+This can help diagnose event capture, delivery failures, retries, and buffer overflow.
+
+## Example application
+
+```python
+from celery import Celery
+from queuelens import QueueLens
+
+app = Celery(
+    "orders",
+    broker="redis://localhost:6379/0",
+    backend="redis://localhost:6379/1",
+)
+
+queuelens = QueueLens()
+queuelens.instrument(app)
+
+
+@app.task
+def process_order(order_id: str):
+    print(f"Processing order: {order_id}")
+    return {"order_id": order_id, "status": "completed"}
+```
+
+Run the worker:
+
+```bash
+celery -A app worker --loglevel=info
+```
+
+## License
+
+See the repository's license information.