rrq 0.4.0__tar.gz → 0.7.0__tar.gz

rrq-0.7.0/.claude/settings.local.json

@@ -0,0 +1,22 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(find:*)",
+      "Bash(grep:*)",
+      "Bash(uv run pytest:*)",
+      "Bash(mkdir:*)",
+      "Bash(mv:*)",
+      "Bash(python -m pytest tests/test_worker.py::test_worker_handles_job_failure_max_retries_dlq -xvs)",
+      "Bash(rg:*)",
+      "Bash(uv run:*)",
+      "Bash(ruff format:*)",
+      "Bash(ruff check:*)",
+      "Bash(uv add:*)",
+      "Bash(sed:*)",
+      "Bash(rm:*)",
+      "Bash(python:*)",
+      "Bash(true)"
+    ],
+    "deny": []
+  }
+}

rrq-0.7.0/.github/workflows/ci.yml

@@ -0,0 +1,37 @@
+name: CI
+
+on:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    services:
+      redis:
+        image: redis:7
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install uv CLI
+        run: |
+          python -m pip install --upgrade pip uv
+
+      - name: Sync dependencies
+        run: uv sync --extra dev
+
+      - name: Run tests
+        run: uv run pytest --disable-warnings -q --maxfail=1

rrq-0.7.0/CLAUDE.md

@@ -0,0 +1,115 @@
+---
+description: RRQ Development Guidelines
+globs: ["**/*", "!tests/**/*"]
+alwaysApply: true
+---
+# RRQ Development Guidelines
+
+## Project Overview
+
+**Important**: Also refer to:
+- `tests/CLAUDE.md` - Testing guidelines
+- `README.md` - Project setup and overview
+
+## Quick Start Commands
+```bash
+# Backend
+uv run pytest             # Running tests
+ruff check                # Lint backend code
+ruff format               # Format backend code
+
+# Package Management
+uv add package_name        # Add dependency
+uv sync --extra dev        # Sync dependencies
+```
+
+## Code References Format
+**CRITICAL**: Always use VS Code clickable format for code references:
+- `app/api/users.py:45` - Single line
+- `app/models/job.py:120-135` - Line range
+- Always use paths relative to project root
+
+### Python Code Style
+- Python 3.10+ with comprehensive type hints
+- Double quotes for strings
+- Max line length: 88 characters
+- PEP 8 naming: `snake_case` functions, `PascalCase` classes
+- Pydantic V2 for data validation
+- Import order: stdlib → third-party → local
+- Docstrings for public interfaces
+- Type annotations for all function signatures
+
+### Code Quality Practices
+- Early returns over nested conditions
+- Small, focused functions (single responsibility)
+- Descriptive variable names
+- Comprehensive error handling with specific exceptions
+- Consistent async/await patterns
+- Use `match/case` for complex conditionals
+
+## Self-Review Checklist for Large Changes
+
+Before submitting significant backend changes:
+
+### Code Quality
+- [ ] All functions have type hints
+- [ ] Complex logic is well-commented
+- [ ] No debug prints or commented code
+- [ ] Follows existing patterns in codebase
+- [ ] Proper error handling throughout
+- [ ] Idiomatic code using Python, Asyncio, and Pydantic best practices
+
+### Testing
+- [ ] Unit tests for new functionality
+- [ ] Integration tests for API changes
+- [ ] All tests pass with `--warnings-as-errors`
+- [ ] Edge cases covered
+- [ ] Mocked external dependencies
+
+### Performance
+- [ ] Database queries are optimized (N+1 prevention)
+- [ ] Async operations are properly awaited
+- [ ] No blocking I/O in async contexts
+- [ ] Background jobs for heavy operations
+
+### Security
+- [ ] Input validation on all endpoints
+- [ ] No sensitive data in logs
+- [ ] SQL injection prevention
+
+### Documentation
+- [ ] API endpoints documented
+- [ ] Complex functions have docstrings
+- [ ] Schema changes documented
+- [ ] Migration files created if needed
+
+## Linting and Pre-commit
+
+**Always run before committing:**
+```bash
+# Format and lint
+ruff format 
+ruff check  --fix
+
+```
+
+## Important Development Rules
+
+1. **Never commit broken tests** - Fix root causes
+2. **Ask before large changes** - Especially cross-domain
+3. **Follow existing patterns** - Check similar code first
+4. **Quality over speed** - Do it right the first time
+5. **Security first** - Never expose sensitive data
+
+## Performance Considerations
+- Profile before optimizing
+- Use async correctly (no sync in async)
+- Cache expensive operations
+- Paginate large result sets
+- Monitor query performance
+
+## Debugging Tips
+- Use `uv run pytest --maxfail=1` for failing tests
+- Use debugger with `import pdb; pdb.set_trace()`
+
+Remember: If unsure about implementation, check existing code patterns first!

{rrq-0.4.0 → rrq-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rrq
-Version: 0.4.0
+Version: 0.7.0
 Summary: RRQ is a Python library for creating reliable job queues using Redis and asyncio
 Project-URL: Homepage, https://github.com/getresq/rrq
 Project-URL: Bug Tracker, https://github.com/getresq/rrq/issues
@@ -8,16 +8,18 @@ Author-email: Mazdak Rezvani <mazdak@me.com>
 License-File: LICENSE
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: System :: Distributed Computing
 Classifier: Topic :: System :: Monitoring
-Requires-Python: >=3.11
+Requires-Python: >=3.10
 Requires-Dist: click>=8.1.3
 Requires-Dist: pydantic-settings>=2.9.1
 Requires-Dist: pydantic>=2.11.4
 Requires-Dist: redis[hiredis]<6,>=4.2.0
+Requires-Dist: rich>=14.0.0
 Requires-Dist: watchfiles>=0.19.0
 Provides-Extra: dev
 Requires-Dist: pytest-asyncio>=1.0.0; extra == 'dev'
@@ -29,18 +31,34 @@ Description-Content-Type: text/markdown
 
 RRQ is a Python library for creating reliable job queues using Redis and `asyncio`, inspired by [ARQ (Async Redis Queue)](https://github.com/samuelcolvin/arq). It focuses on providing at-least-once job processing semantics with features like automatic retries, job timeouts, dead-letter queues, and graceful worker shutdown.
 
+## 🆕 What's New in v0.7.0
+
+- **Comprehensive CLI Tools**: 15+ new commands for monitoring, debugging, and management
+- **Real-time Monitoring Dashboard**: Interactive dashboard with `rrq monitor`
+- **Enhanced DLQ Management**: Sophisticated filtering and requeuing capabilities
+- **Python 3.10 Support**: Expanded compatibility from Python 3.11+ to 3.10+
+- **Bug Fixes**: Critical fix for unique job enqueue failures with proper deferral
+
+## Requirements
+
+- Python 3.10 or higher
+- Redis 5.0 or higher
+- asyncio-compatible environment
+
 ## Key Features
 
 *   **At-Least-Once Semantics**: Uses Redis locks to ensure a job is processed by only one worker at a time. If a worker crashes or shuts down mid-processing, the lock expires, and the job *should* be re-processed (though re-queueing on unclean shutdown isn't implemented here yet - graceful shutdown *does* re-queue).
 *   **Automatic Retries with Backoff**: Jobs that fail with standard exceptions are automatically retried based on `max_retries` settings, using exponential backoff for delays.
 *   **Explicit Retries**: Handlers can raise `RetryJob` to control retry attempts and delays.
 *   **Job Timeouts**: Jobs exceeding their configured timeout (`job_timeout_seconds` or `default_job_timeout_seconds`) are terminated and moved to the DLQ.
-*   **Dead Letter Queue (DLQ)**: Jobs that fail permanently (max retries reached, fatal error, timeout) are moved to a DLQ list in Redis for inspection.
+*   **Dead Letter Queue (DLQ)**: Jobs that fail permanently (max retries reached, fatal error, timeout) are moved to a single global DLQ list in Redis. Each failed job retains its original queue information, allowing for filtered inspection and selective requeuing.
 *   **Job Uniqueness**: The `_unique_key` parameter in `enqueue` prevents duplicate jobs based on a custom key within a specified TTL.
 *   **Graceful Shutdown**: Workers listen for SIGINT/SIGTERM and attempt to finish active jobs within a grace period before exiting. Interrupted jobs are re-queued.
 *   **Worker Health Checks**: Workers periodically update a health key in Redis with a TTL, allowing monitoring systems to track active workers.
 *   **Deferred Execution**: Jobs can be scheduled to run at a future time using `_defer_by` or `_defer_until`.
 *   **Cron Jobs**: Periodic jobs can be defined in `RRQSettings.cron_jobs` using a simple cron syntax.
+*   **Comprehensive Monitoring**: Built-in CLI tools for monitoring queues, inspecting jobs, and debugging with real-time dashboards and beautiful table output.
+*   **Development Tools**: Debug commands for generating test data, stress testing, and cleaning up development environments.
 
     - Using deferral with a specific `_job_id` will effectively reschedule the job associated with that ID to the new time, overwriting its previous definition and score. It does not create multiple distinct scheduled jobs with the same ID.
 
@@ -196,21 +214,21 @@ cron_jobs = [
         args=["temp_files"],
         kwargs={"max_age_days": 7}
     ),
-    
+
     # Weekly report every Monday at 9 AM
     CronJob(
         function_name="generate_weekly_report",
         schedule="0 9 * * mon",
         unique=True  # Prevent duplicate reports if worker restarts
     ),
-    
+
     # Health check every 15 minutes on a specific queue
     CronJob(
         function_name="system_health_check",
         schedule="*/15 * * * *",
         queue_name="monitoring"
     ),
-    
+
     # Backup database every night at 1 AM
     CronJob(
         function_name="backup_database",
@@ -258,25 +276,149 @@ rrq_settings.job_registry = job_registry
 
 **Note:** Cron jobs are automatically enqueued by the worker when they become due. The worker checks for due cron jobs every 30 seconds and enqueues them as regular jobs to be processed.
 
+## Dead Letter Queue (DLQ) Management
+
+RRQ uses a single global Dead Letter Queue to store jobs that have failed permanently. Jobs in the DLQ retain their original queue information, allowing for sophisticated filtering and management.
+
+### DLQ Structure
+
+- **Global DLQ**: One DLQ per RRQ instance (configurable via `default_dlq_name`)
+- **Queue Preservation**: Each failed job remembers its original queue name
+- **Filtering**: Jobs can be filtered by original queue, function name, error patterns, and time ranges
+- **Inspection**: Full job details including arguments, errors, and execution timeline
+
+### Common DLQ Workflows
+
+#### Investigating Failures
+```bash
+# Get overall DLQ statistics
+rrq dlq stats
+
+# List recent failures from a specific queue
+rrq dlq list --queue urgent --limit 10
+
+# Group failures by function
+rrq dlq list --function send_email
+
+# Inspect a specific failed job
+rrq dlq inspect job_abc123
+```
+
+#### Requeuing Failed Jobs
+```bash
+# Preview what would be requeued (dry run)
+rrq dlq requeue --queue urgent --dry-run
+
+# Requeue all failures from urgent queue
+rrq dlq requeue --queue urgent --all
+
+# Requeue specific function failures with limit
+rrq dlq requeue --function send_email --limit 10
+
+# Requeue single job to different queue
+rrq dlq requeue --job-id abc123 --target-queue retry_queue
+```
+
+#### Monitoring DLQ in Real-time
+```bash
+# Monitor includes DLQ statistics panel
+rrq monitor
+
+# Queue stats show DLQ count per original queue
+rrq queue stats
+```
+
 ## Command Line Interface
 
-RRQ provides a command-line interface (CLI) for managing workers and performing health checks:
-
-- **`rrq worker run`** - Run an RRQ worker process.
-  - `--settings` (optional): Specify the Python path to your settings object (e.g., `myapp.worker_config.rrq_settings`). If not provided, it will use the `RRQ_SETTINGS` environment variable or default to a basic `RRQSettings` object.
-  - `--queue` (optional, multiple): Specify queue(s) to poll. Defaults to the `default_queue_name` in settings.
-  - `--burst` (flag): Run the worker in burst mode to process one job or batch and then exit.
-- **`rrq worker watch`** - Run an RRQ worker with auto-restart on file changes.
-  - `--path` (optional): Directory path to watch for changes. Defaults to the current directory.
-  - `--settings` (optional): Same as above.
-  - `--queue` (optional, multiple): Same as above.
-- **`rrq check`** - Perform a health check on active RRQ workers.
-  - `--settings` (optional): Same as above.
-- **`rrq dlq requeue`** - Requeue jobs from the dead letter queue back into a live queue.
-  - `--settings` (optional): Same as above.
-  - `--dlq-name` (optional): Name of the DLQ (without prefix). Defaults to `default_dlq_name` in settings.
-  - `--queue` (optional): Target queue name (without prefix). Defaults to `default_queue_name` in settings.
-  - `--limit` (optional): Maximum number of DLQ jobs to requeue; all if not set.
+RRQ provides a comprehensive command-line interface (CLI) for managing workers, monitoring queues, and debugging.
+
+📖 **[Full CLI Reference Documentation](docs/CLI_REFERENCE.md)**
+
+### Quick Examples
+```bash
+# Use default settings (localhost Redis)
+rrq queue list
+
+# Use custom settings
+rrq queue list --settings myapp.config.rrq_settings
+
+# Use environment variable
+export RRQ_SETTINGS=myapp.config.rrq_settings
+rrq monitor
+
+# Debug workflow
+rrq debug generate-jobs --count 100 --queue urgent
+rrq queue inspect urgent --limit 10
+rrq monitor --queues urgent --refresh 0.5
+
+# DLQ management workflow
+rrq dlq list --queue urgent --limit 10      # List failed jobs from urgent queue
+rrq dlq stats                                # Show DLQ statistics and error patterns
+rrq dlq inspect <job_id>                     # Inspect specific failed job
+rrq dlq requeue --queue urgent --dry-run     # Preview requeue of urgent queue jobs
+rrq dlq requeue --queue urgent --limit 5     # Requeue 5 jobs from urgent queue
+
+# Advanced DLQ filtering and management
+rrq dlq list --function send_email --limit 20          # List failed email jobs
+rrq dlq list --queue urgent --function process_data    # Filter by queue AND function
+rrq dlq requeue --function send_email --all            # Requeue all failed email jobs
+rrq dlq requeue --job-id abc123 --target-queue retry   # Requeue specific job to retry queue
+```
+
+## Performance and Limitations
+
+### Monitoring Performance Considerations
+
+RRQ's monitoring and statistics commands are designed for operational visibility but have some performance considerations for large-scale deployments:
+
+#### Queue Statistics (`rrq queue stats`)
+- **Pending Job Counts**: Very fast, uses Redis `ZCARD` operation
+- **Active/Completed/Failed Counts**: Requires scanning job records in Redis which can be slow for large datasets
+- **Optimization**: Use `--max-scan` parameter to limit scanning (default: 1,000 jobs)
+  ```bash
+  # Fast scan for quick overview
+  rrq queue stats --max-scan 500
+
+  # Complete scan (may be slow)
+  rrq queue stats --max-scan 0
+  ```
+
+#### DLQ Operations (`rrq dlq`)
+- **Job Listing**: Uses batch fetching with Redis pipelines for efficiency
+- **Optimization**: Use `--batch-size` parameter to control memory vs. performance trade-offs
+  ```bash
+  # Smaller batches for memory-constrained environments
+  rrq dlq list --batch-size 50
+
+  # Larger batches for better performance
+  rrq dlq list --batch-size 200
+  ```
+
+#### Real-time Monitoring (`rrq monitor`)
+- **Error Message Truncation**: Newest errors truncated to 50 characters, error patterns to 50 characters for display consistency
+- **DLQ Statistics**: Updates in real-time but may impact Redis performance with very large DLQs
+
+### Full Metrics Requirements
+
+For comprehensive job lifecycle tracking and historical analytics, consider these architectural additions:
+
+1. **Job History Tracking**:
+   - Store completed/failed job summaries in a separate Redis structure or external database
+   - Implement job completion event logging for time-series analytics
+
+2. **Active Job Monitoring**:
+   - Enhanced worker health tracking with job-level visibility
+   - Real-time active job registry for immediate status reporting
+
+3. **Throughput Calculation**:
+   - Time-series data collection for accurate throughput metrics
+   - Queue-specific performance trend tracking
+
+4. **Scalable Statistics**:
+   - Consider Redis Streams or time-series databases for high-frequency job event tracking
+   - Implement sampling strategies for large-scale deployments
+
+The current implementation prioritizes operational simplicity and immediate visibility over comprehensive historical analytics. For production monitoring at scale, complement RRQ's built-in tools with external monitoring systems.
 
 ## Configuration
 
@@ -290,6 +432,48 @@ RRQ can be configured in several ways, with the following precedence:
 
 **Important Note on `job_registry`**: The `job_registry` attribute in your `RRQSettings` object is **critical** for RRQ to function. It must be an instance of `JobRegistry` and is used to register job handlers. Without a properly configured `job_registry`, workers will not know how to process jobs, and most operations will fail. Ensure it is set in your settings object to map job names to their respective handler functions.
 
+### Comprehensive CLI Command System
+- **New modular CLI architecture** with dedicated command modules for better organization
+- **Enhanced monitoring capabilities** with real-time dashboards and beautiful table output
+- **Extensive DLQ management** commands for inspecting, filtering, and requeuing failed jobs
+- **Job lifecycle management** with detailed inspection and control commands
+- **Queue management** with statistics, purging, and migration capabilities
+- **Debug utilities** for development and testing including stress testing and data generation
+
+## 📚 New CLI Commands
+
+### Monitor Commands
+- `rrq monitor` - Real-time dashboard with queue stats, worker health, and DLQ monitoring
+- `rrq monitor workers` - Detailed worker status and health monitoring
+- `rrq monitor jobs` - Active job tracking and monitoring
+
+### DLQ Commands
+- `rrq dlq list` - List failed jobs with filtering by queue, function, and time
+- `rrq dlq stats` - DLQ statistics including error patterns and queue distribution
+- `rrq dlq inspect` - Detailed inspection of failed jobs
+- `rrq dlq requeue` - Requeue failed jobs with dry-run support
+- `rrq dlq purge` - Clean up old failed jobs
+
+### Queue Commands
+- `rrq queue list` - List all queues with job counts
+- `rrq queue stats` - Detailed queue statistics and throughput metrics
+- `rrq queue inspect` - Inspect pending jobs in queues
+- `rrq queue purge` - Purge jobs from queues with safety confirmations
+- `rrq queue migrate` - Move jobs between queues
+
+### Job Commands
+- `rrq job list` - List jobs with status filtering
+- `rrq job inspect` - Detailed job information including timeline
+- `rrq job result` - Retrieve job results
+- `rrq job cancel` - Cancel active jobs
+- `rrq job retry` - Manually retry failed jobs
+- `rrq job delete` - Delete job records
+
+### Debug Commands
+- `rrq debug generate-jobs` - Generate test jobs for development
+- `rrq debug stress-test` - Stress test the system
+- `rrq debug cleanup` - Clean up test data
+- `rrq debug redis-info` - Redis server information and diagnostics
 
 ## Core Components
 
@@ -298,4 +482,4 @@ RRQ can be configured in several ways, with the following precedence:
 *   **`JobRegistry` (`registry.py`)**: A simple registry to map string function names (used when enqueuing) to the actual asynchronous handler functions the worker should execute.
 *   **`JobStore` (`store.py`)**: An abstraction layer handling all direct interactions with Redis. It manages job definitions (Hashes), queues (Sorted Sets), processing locks (Strings with TTL), unique job locks, and worker health checks.
 *   **`Job` (`job.py`)**: A Pydantic model representing a job, containing its ID, handler name, arguments, status, retry counts, timestamps, results, etc.
-*   **`JobStatus` (`job.py`)**: An Enum defining the possible states of a job (`PENDING`, `ACTIVE`, `COMPLETED`, `FAILED`, `
+*   **`JobStatus` (`job.py`)**: An Enum defining the possible states of a job (`PENDING`, `ACTIVE`, `COMPLETED`, `FAILED`, `DEFERRED`). `