django-cfg 1.1.61__tar.gz → 1.1.63__tar.gz

{django_cfg-1.1.61 → django_cfg-1.1.63}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-cfg
-Version: 1.1.61
+Version: 1.1.63
 Summary: 🚀 Production-ready Django configuration framework with type-safe settings, smart automation, and modern developer experience
 Project-URL: Homepage, https://github.com/markolofsen/django-cfg
 Project-URL: Documentation, https://django-cfg.readthedocs.io
@@ -173,7 +173,7 @@ python manage.py runserver
 - ✅ **Newsletter campaigns** with email tracking
 - ✅ **Lead management** system with CRM integration
 - ✅ **Multi-database routing** and connection pooling
-- ✅ **Background task processing** with Dramatiq
+- ✅ **Background task processing** with production-ready Dramatiq integration
 - ✅ **Webhook testing** with built-in ngrok integration
 - ✅ **Type-safe configuration** with full IDE support
 
@@ -413,8 +413,137 @@ Automatic dev/staging/production detection with appropriate defaults.
 ### 🌐 **Built-in Ngrok Integration**
 Instant webhook testing with zero-config ngrok tunnels for development.
 
-### 🔄 **Background Task Processing**
-Built-in Dramatiq integration for reliable background job processing with Redis.
+### 🔄 **Background Task Processing** %%PRIORITY:HIGH%%
+
+Django-CFG includes a complete Dramatiq integration for reliable background job processing with automatic Redis configuration, worker management, and task monitoring.
+
+#### Features
+- **🚀 Zero Configuration** - Automatic Redis broker setup and queue management
+- **⚡ Smart Worker Management** - Built-in `rundramatiq` command with Django settings integration
+- **🔧 Type-Safe Task Definition** - Full IDE support for task creation and scheduling
+- **📊 Task Monitoring** - Built-in commands for queue status and task management
+- **🐳 Docker Ready** - Pre-configured Docker containers for production deployment
+- **🛡️ Production Tested** - Battle-tested configuration used in production environments
+
+#### Quick Setup
+```python
+from django_cfg import DjangoConfig
+
+class MyConfig(DjangoConfig):
+    project_name: str = "My App"
+    
+    # Dramatiq configuration (automatic)
+    redis_url: str = "redis://localhost:6379/2"  # Separate DB for tasks
+    
+    # Optional: Custom task settings
+    dramatiq: DramatiqConfig = DramatiqConfig(
+        processes=2,           # Worker processes
+        threads=4,            # Threads per process
+        redis_db=2,           # Redis database for tasks
+        queues=["default", "high", "low"]  # Available queues
+    )
+
+config = MyConfig()
+```
+
+#### Task Definition
+```python
+import dramatiq
+from django_cfg.modules.django_tasks import get_broker
+
+# Tasks are automatically discovered from your apps
+@dramatiq.actor(queue_name="high", max_retries=3)
+def process_document(document_id: str) -> dict:
+    """Process document asynchronously with full Django context."""
+    from myapp.models import Document
+    
+    document = Document.objects.get(id=document_id)
+    # Your processing logic here
+    document.status = "processed"
+    document.save()
+    
+    return {"status": "completed", "document_id": document_id}
+
+# Queue the task
+process_document.send(document_id="123")
+```
+
+#### Management Commands
+```bash
+# Start Dramatiq workers with Django settings
+python manage.py rundramatiq --processes 4 --threads 8
+
+# Monitor task queues and status
+python manage.py task_status --queue high
+
+# Clear specific queues
+python manage.py task_clear --queue default
+
+# Test task processing pipeline
+python manage.py test_tasks
+```
+
+#### Docker Integration
+```yaml
+# docker-compose.yml (automatically generated)
+services:
+  app-dramatiq:
+    build: .
+    command: ["python", "manage.py", "rundramatiq"]
+    environment:
+      - DRAMATIQ_PROCESSES=2
+      - DRAMATIQ_THREADS=4
+      - DRAMATIQ_QUEUES=default,high,low
+    depends_on:
+      - redis
+      - postgres
+```
+
+#### Production Features
+- **🔄 Automatic Restart** - Workers restart on code changes in development
+- **📈 Scaling** - Easy horizontal scaling with multiple worker containers
+- **🛡️ Error Handling** - Built-in retry logic and dead letter queues
+- **📊 Monitoring** - Integration with Django admin for task monitoring
+- **⚡ Performance** - Optimized Redis configuration for high throughput
+
+#### Real-World Example
+```python
+# Document processing pipeline
+@dramatiq.actor(queue_name="knowledge", max_retries=2)
+def process_document_async(document_id: str) -> dict:
+    """Complete document processing with chunking and embeddings."""
+    try:
+        document = Document.objects.get(id=document_id)
+        
+        # Step 1: Chunk document
+        chunks = create_document_chunks(document)
+        
+        # Step 2: Generate embeddings
+        for chunk in chunks:
+            generate_embeddings.send(chunk.id)
+        
+        # Step 3: Optimize database
+        optimize_document_embeddings.send(document_id)
+        
+        return {"status": "completed", "chunks_created": len(chunks)}
+        
+    except Exception as e:
+        logger.error(f"Document processing failed: {e}")
+        return {"status": "failed", "error": str(e)}
+
+# Triggered automatically via Django signals
+@receiver(post_save, sender=Document)
+def queue_document_processing(sender, instance, created, **kwargs):
+    if created:
+        process_document_async.send(str(instance.id))
+```
+
+**Perfect for:**
+- 📄 **Document Processing** - PDF parsing, text extraction, embeddings
+- 📧 **Email Campaigns** - Bulk email sending with delivery tracking
+- 🔄 **Data Synchronization** - API integrations and data imports
+- 📊 **Report Generation** - Heavy computational tasks
+- 🧹 **Cleanup Tasks** - Database maintenance and optimization
 
 ---
 
@@ -444,6 +573,7 @@ Django-CFG includes powerful management commands for development and operations:
 | **`rundramatiq`** | Run Dramatiq background task workers | `python manage.py rundramatiq --processes 4` |
 | **`task_status`** | Show Dramatiq task status and queues | `python manage.py task_status --queue high` |
 | **`task_clear`** | Clear Dramatiq queues | `python manage.py task_clear --queue default` |
+| **`test_tasks`** | Test Dramatiq task processing pipeline | `python manage.py test_tasks --document-id 123` |
 | **`tree`** | Display Django project structure | `python manage.py tree --depth 3 --include-docs` |
 | **`validate_config`** | Deep validation of all settings | `python manage.py validate_config --strict` |
 
@@ -955,6 +1085,17 @@ class ProductionConfig(DjangoConfig):
     enable_newsletter: bool = True  # Email marketing, campaigns, tracking & analytics
     enable_leads: bool = True       # Lead capture, CRM integration, source tracking
     
+    # === Background Task Processing ===
+    redis_url: str = "redis://redis:6379/2"  # Separate DB for Dramatiq tasks
+    dramatiq: DramatiqConfig = DramatiqConfig(
+        processes=4,              # Production worker processes
+        threads=8,               # Threads per process for I/O bound tasks
+        redis_db=2,              # Dedicated Redis DB for task queues
+        queues=["default", "high", "low", "knowledge", "email"],  # Production queues
+        max_retries=3,           # Default retry attempts
+        max_age=3600000,         # 1 hour max task age (milliseconds)
+    )
+    
     # === Multi-Zone API ===
     revolution: RevolutionConfig = RevolutionConfig(
         api_prefix="api/v2",

{django_cfg-1.1.61 → django_cfg-1.1.63}/README.md

@@ -47,7 +47,7 @@ python manage.py runserver
 - ✅ **Newsletter campaigns** with email tracking
 - ✅ **Lead management** system with CRM integration
 - ✅ **Multi-database routing** and connection pooling
-- ✅ **Background task processing** with Dramatiq
+- ✅ **Background task processing** with production-ready Dramatiq integration
 - ✅ **Webhook testing** with built-in ngrok integration
 - ✅ **Type-safe configuration** with full IDE support
 
@@ -287,8 +287,137 @@ Automatic dev/staging/production detection with appropriate defaults.
 ### 🌐 **Built-in Ngrok Integration**
 Instant webhook testing with zero-config ngrok tunnels for development.
 
-### 🔄 **Background Task Processing**
-Built-in Dramatiq integration for reliable background job processing with Redis.
+### 🔄 **Background Task Processing** %%PRIORITY:HIGH%%
+
+Django-CFG includes a complete Dramatiq integration for reliable background job processing with automatic Redis configuration, worker management, and task monitoring.
+
+#### Features
+- **🚀 Zero Configuration** - Automatic Redis broker setup and queue management
+- **⚡ Smart Worker Management** - Built-in `rundramatiq` command with Django settings integration
+- **🔧 Type-Safe Task Definition** - Full IDE support for task creation and scheduling
+- **📊 Task Monitoring** - Built-in commands for queue status and task management
+- **🐳 Docker Ready** - Pre-configured Docker containers for production deployment
+- **🛡️ Production Tested** - Battle-tested configuration used in production environments
+
+#### Quick Setup
+```python
+from django_cfg import DjangoConfig
+
+class MyConfig(DjangoConfig):
+    project_name: str = "My App"
+    
+    # Dramatiq configuration (automatic)
+    redis_url: str = "redis://localhost:6379/2"  # Separate DB for tasks
+    
+    # Optional: Custom task settings
+    dramatiq: DramatiqConfig = DramatiqConfig(
+        processes=2,           # Worker processes
+        threads=4,            # Threads per process
+        redis_db=2,           # Redis database for tasks
+        queues=["default", "high", "low"]  # Available queues
+    )
+
+config = MyConfig()
+```
+
+#### Task Definition
+```python
+import dramatiq
+from django_cfg.modules.django_tasks import get_broker
+
+# Tasks are automatically discovered from your apps
+@dramatiq.actor(queue_name="high", max_retries=3)
+def process_document(document_id: str) -> dict:
+    """Process document asynchronously with full Django context."""
+    from myapp.models import Document
+    
+    document = Document.objects.get(id=document_id)
+    # Your processing logic here
+    document.status = "processed"
+    document.save()
+    
+    return {"status": "completed", "document_id": document_id}
+
+# Queue the task
+process_document.send(document_id="123")
+```
+
+#### Management Commands
+```bash
+# Start Dramatiq workers with Django settings
+python manage.py rundramatiq --processes 4 --threads 8
+
+# Monitor task queues and status
+python manage.py task_status --queue high
+
+# Clear specific queues
+python manage.py task_clear --queue default
+
+# Test task processing pipeline
+python manage.py test_tasks
+```
+
+#### Docker Integration
+```yaml
+# docker-compose.yml (automatically generated)
+services:
+  app-dramatiq:
+    build: .
+    command: ["python", "manage.py", "rundramatiq"]
+    environment:
+      - DRAMATIQ_PROCESSES=2
+      - DRAMATIQ_THREADS=4
+      - DRAMATIQ_QUEUES=default,high,low
+    depends_on:
+      - redis
+      - postgres
+```
+
+#### Production Features
+- **🔄 Automatic Restart** - Workers restart on code changes in development
+- **📈 Scaling** - Easy horizontal scaling with multiple worker containers
+- **🛡️ Error Handling** - Built-in retry logic and dead letter queues
+- **📊 Monitoring** - Integration with Django admin for task monitoring
+- **⚡ Performance** - Optimized Redis configuration for high throughput
+
+#### Real-World Example
+```python
+# Document processing pipeline
+@dramatiq.actor(queue_name="knowledge", max_retries=2)
+def process_document_async(document_id: str) -> dict:
+    """Complete document processing with chunking and embeddings."""
+    try:
+        document = Document.objects.get(id=document_id)
+        
+        # Step 1: Chunk document
+        chunks = create_document_chunks(document)
+        
+        # Step 2: Generate embeddings
+        for chunk in chunks:
+            generate_embeddings.send(chunk.id)
+        
+        # Step 3: Optimize database
+        optimize_document_embeddings.send(document_id)
+        
+        return {"status": "completed", "chunks_created": len(chunks)}
+        
+    except Exception as e:
+        logger.error(f"Document processing failed: {e}")
+        return {"status": "failed", "error": str(e)}
+
+# Triggered automatically via Django signals
+@receiver(post_save, sender=Document)
+def queue_document_processing(sender, instance, created, **kwargs):
+    if created:
+        process_document_async.send(str(instance.id))
+```
+
+**Perfect for:**
+- 📄 **Document Processing** - PDF parsing, text extraction, embeddings
+- 📧 **Email Campaigns** - Bulk email sending with delivery tracking
+- 🔄 **Data Synchronization** - API integrations and data imports
+- 📊 **Report Generation** - Heavy computational tasks
+- 🧹 **Cleanup Tasks** - Database maintenance and optimization
 
 ---
 
@@ -318,6 +447,7 @@ Django-CFG includes powerful management commands for development and operations:
 | **`rundramatiq`** | Run Dramatiq background task workers | `python manage.py rundramatiq --processes 4` |
 | **`task_status`** | Show Dramatiq task status and queues | `python manage.py task_status --queue high` |
 | **`task_clear`** | Clear Dramatiq queues | `python manage.py task_clear --queue default` |
+| **`test_tasks`** | Test Dramatiq task processing pipeline | `python manage.py test_tasks --document-id 123` |
 | **`tree`** | Display Django project structure | `python manage.py tree --depth 3 --include-docs` |
 | **`validate_config`** | Deep validation of all settings | `python manage.py validate_config --strict` |
 
@@ -829,6 +959,17 @@ class ProductionConfig(DjangoConfig):
     enable_newsletter: bool = True  # Email marketing, campaigns, tracking & analytics
     enable_leads: bool = True       # Lead capture, CRM integration, source tracking
     
+    # === Background Task Processing ===
+    redis_url: str = "redis://redis:6379/2"  # Separate DB for Dramatiq tasks
+    dramatiq: DramatiqConfig = DramatiqConfig(
+        processes=4,              # Production worker processes
+        threads=8,               # Threads per process for I/O bound tasks
+        redis_db=2,              # Dedicated Redis DB for task queues
+        queues=["default", "high", "low", "knowledge", "email"],  # Production queues
+        max_retries=3,           # Default retry attempts
+        max_age=3600000,         # 1 hour max task age (milliseconds)
+    )
+    
     # === Multi-Zone API ===
     revolution: RevolutionConfig = RevolutionConfig(
         api_prefix="api/v2",

{django_cfg-1.1.61 → django_cfg-1.1.63}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "django-cfg"
-version = "1.1.61"
+version = "1.1.63"
 description = "🚀 Production-ready Django configuration framework with type-safe settings, smart automation, and modern developer experience"
 readme = "README.md"
 license = {text = "MIT"}

{django_cfg-1.1.61 → django_cfg-1.1.63}/src/django_cfg/__init__.py

@@ -38,7 +38,7 @@ default_app_config = "django_cfg.apps.DjangoCfgConfig"
 from typing import TYPE_CHECKING
 
 # Version information
-__version__ = "1.1.60"
+__version__ = "1.1.63"
 __author__ = "Unrealos Team"
 __email__ = "info@unrealos.com"
 __license__ = "MIT"

django_cfg-1.1.63/src/django_cfg/management/commands/rundramatiq.py

@@ -0,0 +1,241 @@
+"""
+Django management command for running Dramatiq workers.
+
+Based on django_dramatiq.management.commands.rundramatiq with Django-CFG integration.
+Simple, clean, and working approach.
+"""
+
+import argparse
+import importlib
+import multiprocessing
+import os
+import sys
+
+from django.apps import apps
+from django.conf import settings
+from django.core.management.base import BaseCommand
+from django.utils.module_loading import module_has_submodule
+
+from django_cfg.modules.django_tasks import get_task_service
+
+
+# Default values
+NPROCS = multiprocessing.cpu_count()
+NTHREADS = 8
+
+
+class Command(BaseCommand):
+    help = "Run Dramatiq workers with Django-CFG configuration."
+
+    def add_arguments(self, parser):
+        parser.formatter_class = argparse.ArgumentDefaultsHelpFormatter
+        
+        parser.add_argument(
+            "--processes", "-p",
+            default=NPROCS,
+            type=int,
+            help="The number of processes to run",
+        )
+        parser.add_argument(
+            "--threads", "-t",
+            default=NTHREADS,
+            type=int,
+            help="The number of threads per process to use",
+        )
+        parser.add_argument(
+            "--queues", "-Q",
+            nargs="*",
+            type=str,
+            help="Listen to a subset of queues, or all when empty",
+        )
+        parser.add_argument(
+            "--watch",
+            dest="watch_dir",
+            help="Reload workers when changes are detected in the given directory",
+        )
+        parser.add_argument(
+            "--pid-file",
+            type=str,
+            help="Write the PID of the master process to this file",
+        )
+        parser.add_argument(
+            "--log-file",
+            type=str,
+            help="Write all logs to a file, or stderr when empty",
+        )
+        parser.add_argument(
+            "--worker-shutdown-timeout", 
+            type=int, 
+            default=600000,
+            help="Timeout for worker shutdown, in milliseconds"
+        )
+        parser.add_argument(
+            "--dry-run",
+            action="store_true",
+            help="Show configuration without starting workers",
+        )
+
+    def handle(self, watch_dir, processes, threads, verbosity, queues, 
+               pid_file, log_file, worker_shutdown_timeout, dry_run, **options):
+        
+        # Get task service and validate
+        task_service = get_task_service()
+        if not task_service.is_enabled():
+            self.stdout.write(
+                self.style.ERROR("Task system is not enabled in Django-CFG configuration")
+            )
+            return
+        
+        # Discover task modules
+        tasks_modules = self._discover_tasks_modules()
+        
+        # Show configuration info
+        self.stdout.write(self.style.SUCCESS("Dramatiq Worker Configuration:"))
+        self.stdout.write(f"Processes: {processes}")
+        self.stdout.write(f"Threads: {threads}")
+        if queues:
+            self.stdout.write(f"Queues: {', '.join(queues)}")
+        else:
+            self.stdout.write("Queues: all")
+        
+        self.stdout.write(f"\nDiscovered task modules:")
+        for module in tasks_modules:
+            self.stdout.write(f"  - {module}")
+        
+        # If dry run, show command and exit
+        if dry_run:
+            executable_name = "dramatiq"
+            process_args = [
+                executable_name,
+                "--processes", str(processes),
+                "--threads", str(threads),
+                "--worker-shutdown-timeout", str(worker_shutdown_timeout),
+            ]
+            
+            if watch_dir:
+                process_args.extend(["--watch", watch_dir])
+            
+            verbosity_args = ["-v"] * (verbosity - 1)
+            process_args.extend(verbosity_args)
+            
+            if queues:
+                process_args.extend(["--queues", *queues])
+            
+            if pid_file:
+                process_args.extend(["--pid-file", pid_file])
+            
+            if log_file:
+                process_args.extend(["--log-file", log_file])
+            
+            process_args.extend(tasks_modules)
+            
+            self.stdout.write(f"\nCommand that would be executed:")
+            self.stdout.write(f'  {" ".join(process_args)}')
+            return
+        
+        # Show startup info
+        self.stdout.write(self.style.SUCCESS("\nStarting Dramatiq workers..."))
+        
+        # Build dramatiq command
+        executable_name = "dramatiq"
+        executable_path = self._resolve_executable(executable_name)
+        
+        # Build process arguments exactly like django_dramatiq
+        process_args = [
+            executable_name,
+            "--processes", str(processes),
+            "--threads", str(threads),
+            "--worker-shutdown-timeout", str(worker_shutdown_timeout),
+        ]
+        
+        # Add watch directory if specified
+        if watch_dir:
+            process_args.extend(["--watch", watch_dir])
+        
+        # Add verbosity
+        verbosity_args = ["-v"] * (verbosity - 1)
+        process_args.extend(verbosity_args)
+        
+        # Add queues if specified
+        if queues:
+            process_args.extend(["--queues", *queues])
+        
+        # Add PID file if specified
+        if pid_file:
+            process_args.extend(["--pid-file", pid_file])
+        
+        # Add log file if specified
+        if log_file:
+            process_args.extend(["--log-file", log_file])
+        
+        # Add task modules (broker module first, then discovered modules)
+        process_args.extend(tasks_modules)
+        
+        self.stdout.write(f'Running dramatiq: "{" ".join(process_args)}"\n')
+        
+        # Ensure DJANGO_SETTINGS_MODULE is set for worker processes
+        if not os.environ.get('DJANGO_SETTINGS_MODULE'):
+            if hasattr(settings, 'SETTINGS_MODULE'):
+                os.environ['DJANGO_SETTINGS_MODULE'] = settings.SETTINGS_MODULE
+            else:
+                # Try to detect from manage.py or current settings
+                import django
+                from django.conf import settings as django_settings
+                if hasattr(django_settings, '_wrapped') and hasattr(django_settings._wrapped, '__module__'):
+                    module_name = django_settings._wrapped.__module__
+                    os.environ['DJANGO_SETTINGS_MODULE'] = module_name
+                else:
+                    self.stdout.write(
+                        self.style.WARNING("Could not detect DJANGO_SETTINGS_MODULE")
+                    )
+        
+        # Use os.execvp like django_dramatiq to preserve environment
+        if sys.platform == "win32":
+            import subprocess
+            command = [executable_path] + process_args[1:]
+            sys.exit(subprocess.run(command))
+        
+        os.execvp(executable_path, process_args)
+
+    def _discover_tasks_modules(self):
+        """Discover task modules like django_dramatiq does."""
+        # Always include our broker setup module first
+        tasks_modules = ["django_cfg.modules.dramatiq_setup"]
+        
+        # Get task service for configuration
+        task_service = get_task_service()
+        
+        # Try to get task modules from Django-CFG config
+        if task_service.config and task_service.config.auto_discover_tasks:
+            discovered = task_service.discover_tasks()
+            for module_name in discovered:
+                self.stdout.write(f"Discovered tasks module: '{module_name}'")
+                tasks_modules.append(module_name)
+        
+        # Fallback: use django_dramatiq discovery logic
+        if len(tasks_modules) == 1:  # Only broker module found
+            task_module_names = getattr(settings, "DRAMATIQ_AUTODISCOVER_MODULES", ("tasks",))
+            
+            for app_config in apps.get_app_configs():
+                for task_module in task_module_names:
+                    if module_has_submodule(app_config.module, task_module):
+                        module_name = f"{app_config.name}.{task_module}"
+                        try:
+                            importlib.import_module(module_name)
+                            self.stdout.write(f"Discovered tasks module: '{module_name}'")
+                            tasks_modules.append(module_name)
+                        except ImportError:
+                            # Module exists but has import errors, skip it
+                            pass
+        
+        return tasks_modules
+
+    def _resolve_executable(self, exec_name):
+        """Resolve executable path like django_dramatiq does."""
+        bin_dir = os.path.dirname(sys.executable)
+        if bin_dir:
+            for d in [bin_dir, os.path.join(bin_dir, "Scripts")]:
+                exec_path = os.path.join(d, exec_name)
+                if os.path.isfile(exec_path):
+                    return exec_path
+        return exec_name