abstract-block-dumper 0.0.1__py3-none-any.whl

abstract_block_dumper-0.0.1.dist-info/METADATA

@@ -0,0 +1,311 @@
+Metadata-Version: 2.4
+Name: abstract-block-dumper
+Version: 0.0.1
+Project-URL: Source, https://github.com/bactensor/abstract-block-dumper
+Project-URL: Issue Tracker, https://github.com/bactensor/abstract-block-dumper/issues
+Author-email: Reef Technologies <opensource@reef.pl>
+License: MIT
+Classifier: Framework :: Django
+Classifier: Framework :: Django :: 4.2
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Requires-Dist: bittensor>=9.10.1
+Requires-Dist: celery>=5.5.3
+Requires-Dist: django<6.0,>=3.2
+Requires-Dist: structlog>=25.4.0
+Description-Content-Type: text/markdown
+
+# Abstract Block Dumper
+&nbsp;[![Continuous Integration](https://github.com/bactensor/abstract-block-dumper/workflows/Continuous%20Integration/badge.svg)](https://github.com/bactensor/abstract-block-dumper/actions?query=workflow%3A%22Continuous+Integration%22)&nbsp;[![License](https://img.shields.io/pypi/l/abstract_block_dumper.svg?label=License)](https://pypi.python.org/pypi/abstract_block_dumper)&nbsp;[![python versions](https://img.shields.io/pypi/pyversions/abstract_block_dumper.svg?label=python%20versions)](https://pypi.python.org/pypi/abstract_block_dumper)&nbsp;[![PyPI version](https://img.shields.io/pypi/v/abstract_block_dumper.svg?label=PyPI%20version)](https://pypi.python.org/pypi/abstract_block_dumper)
+
+This package provides a simplified framework for creating block processing tasks in Django applications.
+Define tasks with lambda conditions using the @block_task decorator and run them asynchronously with Celery.
+
+## Implementation Details
+
+### General Workflow:
+Register functions -> detect new blocks -> evaluate conditions -> send to Celery -> execute -> track results -> handle retries.
+
+
+### WorkflowSteps
+1. Register
+- Functions are automatically discovered when the scheduler starts
+- Functions must be located in installed apps in tasks.py or block_tasks.py
+- Functions marked with @block_task decorators are stored in memory registry
+
+2. Detect Blocks
+- Scheduler is running by management command block_tasks
+- Scheduler polls blockchain, finds new blocks, and batches them
+
+3. Plan Tasks
+- For each block, lambda conditions are evaluated against registered functions
+- Tasks are created for matching conditions (with optional multiple argument sets)
+
+4. Queue
+Tasks are sent to Celery with queue and timeout settings from celery_kwargs
+
+5. Execute
+Celery runs the function with block info, capturing results and errors
+
+6. Track
+Task attempts are stored in TaskAttempt model with retry logic and state tracking
+
+
+## Prerequisites
+- Django
+- Celery
+- Redis (for Celery broker and result backend)
+- PostgreSQL (recommended for production)
+
+## Installation
+
+1. Install the package:
+```bash
+pip install abstract_block_dumper
+```
+
+2. Add to your Django `INSTALLED_APPS`:
+```python
+INSTALLED_APPS = [
+    # ... other apps
+    'abstract_block_dumper',
+]
+```
+
+3. Run migrations:
+```bash
+python manage.py migrate
+```
+
+## Usage
+
+### 1. Define Block Processing Tasks
+Create block processing tasks in `tasks.py` or `block_tasks.py` file inside any of your installed Django apps.
+
+### 2. Use Decorators to Register Tasks
+- Use `@block_task` with lambda conditions to create custom block processing tasks
+
+### 3. Start the Block Scheduler
+Run the scheduler to start processing blocks:
+```bash
+$ python manage.py block_tasks
+```
+
+This command will:
+- Automatically discover and register all decorated functions
+- Start polling the blockchain for new blocks
+- Schedule tasks based on your lambda conditions
+
+### 4. Start Celery Workers
+In separate terminals, start Celery workers to execute tasks:
+```bash
+$ celery -A your_project worker --loglevel=info
+```
+
+See examples below:
+
+Use the `@block_task` decorator with lambda conditions to create block processing tasks:
+
+```python
+from abstract_block_dumper.decorators import block_task
+
+
+# Process every block
+@block_task(condition=lambda bn: True)
+def process_every_block(block_number: int):
+    print(f"Processing every block: {block_number}")
+
+# Process every 10 blocks
+@block_task(condition=lambda bn: bn % 10 == 0)
+def process_every_10_blocks(block_number: int):
+    print(f"Processing every 10 blocks: {block_number}")
+
+# Process with multiple netuids
+@block_task(
+    condition=lambda bn, netuid: bn % 100 == 0,
+    args=[{"netuid": 1}, {"netuid": 3}, {"netuid": 22}],
+    backfilling_lookback=300,
+    celery_kwargs={"queue": "high-priority"}
+)
+def process_multi_netuid_task(block_number: int, netuid: int):
+    print(f"Processing block {block_number} for netuid: {netuid}")
+```
+
+
+## Maintenance Tasks
+
+### Cleanup Old Task Attempts
+
+The framework provides a maintenance task to clean up old task records and maintain database performance:
+
+```python
+from abstract_block_dumper.tasks import cleanup_old_tasks
+
+# Delete tasks older than 7 days (default)
+cleanup_old_tasks.delay()
+
+# Delete tasks older than 30 days
+cleanup_old_tasks.delay(days=30)
+```
+
+This task deletes all succeeded or unrecoverable failed tasks older than the specified number of days. It never deletes tasks with PENDING or RUNNING status to ensure ongoing work is preserved.
+
+#### Running the Cleanup Task
+
+**Option 1: Manual Execution**
+```bash
+# Using Django shell
+python manage.py shell -c "from abstract_block_dumper.tasks import cleanup_old_tasks; cleanup_old_tasks.delay()"
+```
+
+**Option 2: Cron Job (Recommended - once per day)**
+```bash
+# Add to crontab (daily at 2 AM)
+0 2 * * * cd /path/to/your/project && python manage.py shell -c "from abstract_block_dumper.tasks import cleanup_old_tasks; cleanup_old_tasks.delay()"
+```
+
+**Option 3: Celery Beat (Automated Scheduling)**
+
+Add this to your Django `settings.py`:
+
+```python
+from celery.schedules import crontab
+
+CELERY_BEAT_SCHEDULE = {
+    'cleanup-old-tasks': {
+        'task': 'abstract_block_dumper.cleanup_old_tasks',
+        'schedule': crontab(hour=2, minute=0),  # Daily at 2 AM
+        'kwargs': {'days': 7},  # Customize retention period
+    },
+}
+```
+
+Then start the Celery beat scheduler:
+```bash
+celery -A your_project beat --loglevel=info
+```
+
+## Configuration
+
+### Required Django Settings
+
+Add these settings to your Django `settings.py`:
+
+```python
+# Celery Configuration
+CELERY_BROKER_URL = 'redis://localhost:6379/0'
+CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
+
+# Abstract Block Dumper specific settings
+BITTENSOR_NETWORK = 'finney'  # Options: 'finney', 'local', 'testnet', 'mainnet'
+BLOCK_DUMPER_START_FROM_BLOCK = 'current'  # Options: None, 'current', or int
+BLOCK_DUMPER_POLL_INTERVAL = 1  # seconds between polling for new blocks
+BLOCK_TASK_RETRY_BACKOFF = 2  # minutes for retry backoff base
+BLOCK_DUMPER_MAX_ATTEMPTS = 3  # maximum retry attempts
+BLOCK_TASK_MAX_RETRY_DELAY_MINUTES = 1440  # maximum retry delay (24 hours)
+```
+
+### Configuration Options Reference
+
+#### Core Settings
+
+**BITTENSOR_NETWORK** (str, default: `'finney'`) Specifies which [Bittensor network](https://docs.learnbittensor.org/concepts/bittensor-networks) to connect to
+
+**BLOCK_DUMPER_START_FROM_BLOCK** (str|int|None, default: `None`)
+- **Purpose**: Determines the starting block for processing when the scheduler first runs
+- **Valid Values**:
+  - `None`: Resume from the last processed block stored in database
+  - `'current'`: Start from the current blockchain block (skips historical blocks)
+  - `int`: Start from a specific block number (e.g., `1000000`)
+- **Example**: `BLOCK_DUMPER_START_FROM_BLOCK = 'current'`
+- **Performance Impact**: Starting from historical blocks may require significant processing time
+
+#### Scheduler Settings
+
+**BLOCK_DUMPER_POLL_INTERVAL** (int, default: `1`)
+- **Purpose**: Seconds to wait between checking for new blocks
+- **Valid Range**: `1` to `3600` (1 second to 1 hour)
+- **Example**: `BLOCK_DUMPER_POLL_INTERVAL = 5`
+- **Performance Impact**:
+  - Lower values (1-2s): Near real-time processing, higher CPU/network usage
+  - Higher values (10-60s): Reduced load but delayed processing
+  - Very low values (<1s) may cause rate limiting
+
+#### Retry and Error Handling Settings
+
+**BLOCK_DUMPER_MAX_ATTEMPTS** (int, default: `3`)
+- **Purpose**: Maximum number of attempts to retry a failed task before giving up
+- **Valid Range**: `1` to `10`
+- **Example**: `BLOCK_DUMPER_MAX_ATTEMPTS = 5`
+- **Performance Impact**: Higher values increase resilience but may delay failure detection
+
+**BLOCK_TASK_RETRY_BACKOFF** (int, default: `1`)
+- **Purpose**: Base number of minutes for exponential backoff retry delays
+- **Valid Range**: `1` to `60`
+- **Example**: `BLOCK_TASK_RETRY_BACKOFF = 2`
+- **Calculation**: Actual delay = `backoff ** attempt_count` minutes
+  - Attempt 1: 2¹ = 2 minutes
+  - Attempt 2: 2² = 4 minutes  
+  - Attempt 3: 2³ = 8 minutes
+- **Performance Impact**: Lower values retry faster but may overwhelm failing services
+
+**BLOCK_TASK_MAX_RETRY_DELAY_MINUTES** (int, default: `1440`)
+- **Purpose**: Maximum delay (in minutes) between retry attempts, caps exponential backoff
+- **Valid Range**: `1` to `10080` (1 minute to 1 week)
+- **Example**: `BLOCK_TASK_MAX_RETRY_DELAY_MINUTES = 720`  # 12 hours max
+- **Performance Impact**: Prevents extremely long delays while maintaining backoff benefits
+
+
+## Example Project
+
+The repository includes a complete working example in the `example_project/` directory that demonstrates:
+
+- Django application setup with abstract-block-dumper
+- Multiple task types (`@every_block`, `@every_n_blocks` with different configurations)
+- Error handling with a randomly failing task
+- Docker Compose setup with all required services
+- Monitoring with Flower (Celery monitoring tool)
+
+### Running the Example
+
+```bash
+cd example_project
+docker-compose up --build
+```
+
+This starts:
+- **Django application** (http://localhost:8000) - Admin interface (user: `admin`, password: `admin`)
+- **Celery workers** - Execute block processing tasks
+- **Block scheduler** - Monitors blockchain and schedules tasks
+- **Flower monitoring** (http://localhost:5555) - Monitor Celery tasks
+- **Redis & PostgreSQL** - Required services
+
+
+## Development
+
+
+Pre-requisites:
+- [uv](https://docs.astral.sh/uv/)
+- [nox](https://nox.thea.codes/en/stable/)
+- [docker](https://www.docker.com/) and [docker compose plugin](https://docs.docker.com/compose/)
+
+
+Ideally, you should run `nox -t format lint` before every commit to ensure that the code is properly formatted and linted.
+Before submitting a PR, make sure that tests pass as well, you can do so using:
+```
+nox -t check # equivalent to `nox -t format lint test`
+```
+
+If you wish to install dependencies into `.venv` so your IDE can pick them up, you can do so using:
+```
+uv sync --all-extras --dev
+```
+
+### Release process
+
+Run `nox -s make_release -- X.Y.Z` where `X.Y.Z` is the version you're releasing and follow the printed instructions.

abstract_block_dumper-0.0.1.dist-info/RECORD

@@ -0,0 +1,25 @@
+abstract_block_dumper/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/_version.py,sha256=qf6R-J7-UyuABBo8c0HgaquJ8bejVbf07HodXgwAwgQ,704
+abstract_block_dumper/admin.py,sha256=3J3I_QOKFgfMNpTXW-rTQGO_q5Ls6uNuL0FkPVdIsYg,1654
+abstract_block_dumper/apps.py,sha256=DXATdrjsL3T2IletTbKeD6unr8ScLaxg7wz0nAHTAns,215
+abstract_block_dumper/decorators.py,sha256=lV1ueIlEbBNojnXVH5GQiRCbck3-SQgtWOil5OqeTHo,7061
+abstract_block_dumper/discovery.py,sha256=kZlb8y-0ltJE-L-1GLxZ_xlziibY8AjggvHJ9sxsScw,728
+abstract_block_dumper/exceptions.py,sha256=EunFH-H5eXNNkKl2CvHlhZ2wvtdry969Gle-CZc7YM0,315
+abstract_block_dumper/models.py,sha256=l229tar4FdQ52eETLKGeskgkXHWa4ealF6DWbG8M4Mc,2012
+abstract_block_dumper/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/tasks.py,sha256=8ppGWxML3krVdrS_08WnKuCpERRhB_6DIyVEkpYZMrw,2638
+abstract_block_dumper/dal/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/dal/django_dal.py,sha256=unAA4Mt5dBBaUhvyezfyC0VtWMD6Ru79NyjKaOMNNSw,5359
+abstract_block_dumper/dal/memory_registry.py,sha256=rgU2CYGm2MHPgSZefgr-kuLxOtPu5wxINa3Y5ELgMUo,3029
+abstract_block_dumper/management/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/management/commands/block_tasks.py,sha256=dEfFnoZCIIDsrNL5vRPtIDrkpcJk36yev_aoGAScgoQ,758
+abstract_block_dumper/migrations/0001_initial.py,sha256=ImPHC3G6kPkq4Xn_4YVAm4Labh1Xi7PkCRszYRGpTiI,2298
+abstract_block_dumper/migrations/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/services/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+abstract_block_dumper/services/block_processor.py,sha256=4guYwtVYh-N1UewiqVN7xG5pM56adyGe8vPG_kCDmQI,8072
+abstract_block_dumper/services/executor.py,sha256=TDbrtVGiz7GNGJwHYB6ZqqhrrTDDL7JGzxOehpF-QTY,1786
+abstract_block_dumper/services/scheduler.py,sha256=zKY24zSwjcQSVk3wt39GBurSNXkfylWsdV7Mgmv1RO8,3443
+abstract_block_dumper/services/utils.py,sha256=Iqa-9xhNxOCnvSWjGBclOUvmO4qsUhhievUllVh82I4,1286
+abstract_block_dumper-0.0.1.dist-info/METADATA,sha256=g26Qm3r1ZwH5OCCVCndRdjwnn-VY4YdIxwQyGKuVHXA,11022
+abstract_block_dumper-0.0.1.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+abstract_block_dumper-0.0.1.dist-info/RECORD,,

abstract_block_dumper-0.0.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.27.0
+Root-Is-Purelib: true
+Tag: py3-none-any