avtomatika-worker 1.0a2__tar.gz → 1.0b2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. avtomatika_worker-1.0b2/PKG-INFO +610 -0
  2. avtomatika_worker-1.0b2/README.md +583 -0
  3. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/pyproject.toml +13 -4
  4. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker/__init__.py +2 -1
  5. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker/config.py +46 -36
  6. avtomatika_worker-1.0b2/src/avtomatika_worker/s3.py +141 -0
  7. avtomatika_worker-1.0b2/src/avtomatika_worker/task_files.py +97 -0
  8. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker/types.py +4 -0
  9. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker/worker.py +211 -78
  10. avtomatika_worker-1.0b2/src/avtomatika_worker.egg-info/PKG-INFO +610 -0
  11. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker.egg-info/SOURCES.txt +10 -1
  12. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker.egg-info/requires.txt +6 -0
  13. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/tests/test_concurrency_limits.py +11 -2
  14. avtomatika_worker-1.0b2/tests/test_config.py +118 -0
  15. avtomatika_worker-1.0b2/tests/test_dependency_injection.py +117 -0
  16. avtomatika_worker-1.0b2/tests/test_init.py +25 -0
  17. avtomatika_worker-1.0b2/tests/test_parameter_typing.py +168 -0
  18. avtomatika_worker-1.0b2/tests/test_per_orchestrator_token.py +62 -0
  19. avtomatika_worker-1.0b2/tests/test_s3.py +179 -0
  20. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/tests/test_worker_logic.py +21 -19
  21. avtomatika_worker-1.0b2/tests/test_worker_more_logic.py +272 -0
  22. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/tests/test_worker_sdk.py +74 -48
  23. avtomatika_worker-1.0b2/tests/test_wrr_logic.py +63 -0
  24. avtomatika_worker-1.0a2/PKG-INFO +0 -307
  25. avtomatika_worker-1.0a2/README.md +0 -285
  26. avtomatika_worker-1.0a2/src/avtomatika_worker.egg-info/PKG-INFO +0 -307
  27. avtomatika_worker-1.0a2/tests/test_config.py +0 -96
  28. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/LICENSE +0 -0
  29. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/setup.cfg +0 -0
  30. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker.egg-info/dependency_links.txt +0 -0
  31. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/src/avtomatika_worker.egg-info/top_level.txt +0 -0
  32. {avtomatika_worker-1.0a2 → avtomatika_worker-1.0b2}/tests/test_types.py +0 -0
@@ -0,0 +1,610 @@
1
+ Metadata-Version: 2.4
2
+ Name: avtomatika-worker
3
+ Version: 1.0b2
4
+ Summary: Worker SDK for the Avtomatika orchestrator.
5
+ Project-URL: Homepage, https://github.com/avtomatika-ai/avtomatika-worker
6
+ Project-URL: Bug Tracker, https://github.com/avtomatika-ai/avtomatika-worker/issues
7
+ Classifier: Development Status :: 4 - Beta
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: License :: OSI Approved :: MIT License
10
+ Classifier: Operating System :: OS Independent
11
+ Requires-Python: >=3.11
12
+ Description-Content-Type: text/markdown
13
+ License-File: LICENSE
14
+ Requires-Dist: aiohttp~=3.13.2
15
+ Requires-Dist: python-json-logger~=4.0.0
16
+ Requires-Dist: aioboto3~=15.5.0
17
+ Requires-Dist: aiofiles~=25.1.0
18
+ Provides-Extra: test
19
+ Requires-Dist: pytest; extra == "test"
20
+ Requires-Dist: pytest-asyncio; extra == "test"
21
+ Requires-Dist: aioresponses; extra == "test"
22
+ Requires-Dist: pytest-mock; extra == "test"
23
+ Requires-Dist: pydantic; extra == "test"
24
+ Provides-Extra: pydantic
25
+ Requires-Dist: pydantic; extra == "pydantic"
26
+ Dynamic: license-file
27
+
28
+ # Avtomatika Worker SDK
29
+
30
+ This is an SDK for creating workers compatible with the **Avtomatika** orchestrator. The SDK handles all the complexity of interacting with the orchestrator, allowing you to focus on writing your business logic.
31
+
32
+ ## Installation
33
+
34
+ ```bash
35
+ pip install avtomatika-worker
36
+ ```
37
+
38
+ For advanced validation features, you can install the SDK with the `pydantic` extra:
39
+ ```bash
40
+ pip install "avtomatika-worker[pydantic]"
41
+ ```
42
+
43
+ ## Configuration
44
+
45
+ The worker is configured entirely through environment variables. Before running your worker, you need to set a few essential variables.
46
+
47
+ - **`WORKER_ID`**: A unique name for your worker instance. If not provided, a random UUID will be generated.
48
+ - **`ORCHESTRATOR_URL`**: The address of the Avtomatika orchestrator.
49
+ - **`WORKER_TOKEN`**: An authentication token to connect to the orchestrator.
50
+
51
+ Here is an example of how you might set them in your shell:
52
+ ```bash
53
+ export WORKER_ID="image-processor-worker-1"
54
+ export ORCHESTRATOR_URL="http://localhost:8080"
55
+ export WORKER_TOKEN="your-secret-token"
56
+ ```
57
+
58
+ A complete list of all available configuration variables can be found in the **Full Configuration Reference** section at the end of this document.
59
+
60
+ ## Programmatic Configuration (Advanced)
61
+
62
+ While using environment variables is the recommended approach, you can also configure the worker programmatically. This is useful for advanced scenarios, such as dynamic configuration or integration into larger applications.
63
+
64
+ The process supports partial configuration. When you create a `WorkerConfig` instance, it **first loads all settings from environment variables**. You can then override specific values in your code before passing the completed config object to the `Worker`.
65
+
66
+ **Note:** The attributes on the `WorkerConfig` object use `UPPERCASE_SNAKE_CASE` to mirror the corresponding environment variables.
67
+
68
+ ### Example of Partial Configuration
69
+
70
+ Let's assume you have an environment variable set for the orchestrator URL:
71
+ ```bash
72
+ export ORCHESTRATOR_URL="http://orchestrator.from.env:8080"
73
+ ```
74
+
75
+ You can then write Python code to override other settings:
76
+ ```python
77
+ import asyncio
78
+ from avtomatika_worker import Worker
79
+ from avtomatika_worker.config import WorkerConfig
80
+
81
+ # 1. Create a config object. It automatically reads ORCHESTRATOR_URL
82
+ # from the environment variables at this step.
83
+ custom_config = WorkerConfig()
84
+
85
+ # 2. Programmatically override or set other attributes.
86
+ custom_config.WORKER_ID = "programmatic-worker-1"
87
+ custom_config.WORKER_TOKEN = "super-secret-token-from-code"
88
+ custom_config.MAX_CONCURRENT_TASKS = 5
89
+
90
+ # 3. Pass the final config object to the Worker.
91
+ # It will use the values from your code (e.g., WORKER_ID)
92
+ # and the values from the environment (e.g., ORCHESTRATORS).
93
+ worker = Worker(
94
+ worker_type="special-cpu-worker",
95
+ config=custom_config
96
+ )
97
+
98
+ @worker.task("do_work")
99
+ async def do_work(params: dict, **kwargs):
100
+ # ...
101
+ return {"status": "success"}
102
+
103
+ if __name__ == "__main__":
104
+ worker.run_with_health_check()
105
+ ```
106
+
107
+
108
+
109
+ ## Quick Start
110
+
111
+ For quick testing and visibility during startup, you can add basic logging configuration to your worker script. This ensures that informational messages, including registration with the orchestrator, are printed to the console.
112
+
113
+ You can configure your worker either via environment variables (recommended for production) or directly in your Python code for quick testing or specialized setups.
114
+
115
+ ### Option 1: Configure via Environment Variables (Recommended)
116
+
117
+ Save the following code as `my_worker.py`:
118
+ ```python
119
+ import asyncio
120
+ import logging # Import logging
121
+ from avtomatika_worker import Worker
122
+
123
+ # Configure basic logging to see worker messages
124
+ logging.basicConfig(level=logging.INFO)
125
+
126
+ # 1. Create a worker instance.
127
+ # The SDK automatically reads the configuration from environment variables.
128
+ worker = Worker(worker_type="image-processing")
129
+
130
+ # 2. Register a task handler using the decorator
131
+ @worker.task("resize_image")
132
+ async def image_resizer(params: dict, **kwargs):
133
+ """
134
+ An example handler that receives task parameters,
135
+ performs the work, and returns the result.
136
+ """
137
+ task_id = kwargs.get("task_id")
138
+ job_id = kwargs.get("job_id")
139
+
140
+ print(f"Task {task_id} (Job: {job_id}): resizing image...")
141
+ print(f"Parameters: {params}")
142
+
143
+ # ... your business logic here ...
144
+ await asyncio.sleep(1) # Simulate I/O-bound work
145
+
146
+ # Return the result
147
+ return {
148
+ "status": "success",
149
+ "data": {
150
+ "resized_path": f"/path/to/resized_{params.get('filename')}"
151
+ }
152
+ }
153
+
154
+ # 3. Run the worker
155
+ if __name__ == "__main__":
156
+ # The SDK will automatically connect to the orchestrator,
157
+ # register itself, and start polling for tasks.
158
+ worker.run_with_health_check()
159
+ ```
160
+
161
+ After setting the required environment variables, you can run your worker.
162
+
163
+ **Example:**
164
+ ```bash
165
+ export WORKER_ID="image-processor-worker-1"
166
+ export ORCHESTRATOR_URL="http://localhost:8080"
167
+ export WORKER_TOKEN="your-secret-token"
168
+
169
+ python my_worker.py
170
+ ```
171
+
172
+ ### Option 2: Configure Programmatically (Alternative)
173
+
174
+ For quick testing or if you prefer to define configuration directly in code for simple examples, you can create and pass a `WorkerConfig` object.
175
+
176
+ Save the following code as `my_worker_programmatic.py`:
177
+ ```python
178
+ import asyncio
179
+ import logging # Import logging
180
+ from avtomatika_worker import Worker
181
+ from avtomatika_worker.config import WorkerConfig # Import WorkerConfig
182
+
183
+ # Configure basic logging to see worker messages
184
+ logging.basicConfig(level=logging.INFO)
185
+
186
+ # 1. Create and configure a WorkerConfig object
187
+ my_config = WorkerConfig()
188
+ my_config.WORKER_ID = "image-processor-worker-1-programmatic"
189
+ my_config.ORCHESTRATOR_URL = "http://localhost:8080"
190
+ my_config.WORKER_TOKEN = "your-secret-token" # Replace with your actual token
191
+
192
+ # 2. Create a worker instance, passing the configured object
193
+ worker = Worker(worker_type="image-processing", config=my_config)
194
+
195
+ # 3. Register a task handler using the decorator
196
+ @worker.task("resize_image")
197
+ async def image_resizer(params: dict, **kwargs):
198
+ task_id = kwargs.get("task_id")
199
+ job_id = kwargs.get("job_id")
200
+
201
+ print(f"Task {task_id} (Job: {job_id}): resizing image...")
202
+ print(f"Parameters: {params}")
203
+
204
+ await asyncio.sleep(1)
205
+ return {
206
+ "status": "success",
207
+ "data": {
208
+ "resized_path": f"/path/to/resized_{params.get('filename')}"
209
+ }
210
+ }
211
+
212
+ # 4. Run the worker
213
+ if __name__ == "__main__":
214
+ worker.run_with_health_check()
215
+ ```
216
+
217
+ Run your worker:
218
+ ```bash
219
+ python my_worker_programmatic.py
220
+ ```
221
+
222
+ ## Defining Task Parameters
223
+
224
+ The SDK offers three ways to define and validate the `params` your task handler receives, giving you the flexibility to choose the right tool for your needs.
225
+
226
+ ### 1. Default: `dict`
227
+
228
+ By default, or if you type-hint `params` as a `dict`, you will receive the raw dictionary of parameters sent by the orchestrator. This is simple and requires no extra definitions.
229
+
230
+ ```python
231
+ @worker.task("resize_image")
232
+ async def image_resizer(params: dict, **kwargs):
233
+ width = params.get("width")
234
+ height = params.get("height")
235
+ # ...
236
+ ```
237
+
238
+ ### 2. Structured: `dataclasses`
239
+
240
+ For better structure and IDE autocompletion, you can use Python's built-in `dataclasses`. The SDK will automatically instantiate the dataclass from the incoming parameters. You can access parameters as class attributes.
241
+
242
+ You can also add custom validation logic using the `__post_init__` method. If validation fails, the SDK will automatically catch the `ValueError` and report an `INVALID_INPUT_ERROR` to the orchestrator.
243
+
244
+ ```python
245
+ from dataclasses import dataclass
246
+
247
+ @dataclass
248
+ class ResizeParams:
249
+ width: int
250
+ height: int
251
+
252
+ def __post_init__(self):
253
+ if self.width <= 0 or self.height <= 0:
254
+ raise ValueError("Width and height must be positive.")
255
+
256
+ @worker.task("resize_image")
257
+ async def image_resizer(params: ResizeParams, **kwargs):
258
+ # Access params with dot notation and autocompletion
259
+ print(f"Resizing to {params.width}x{params.height}")
260
+ # ...
261
+ ```
262
+
263
+ ### 3. Validated: `pydantic`
264
+
265
+ For the most robust validation and type coercion, you can use `pydantic`. First, install the necessary extra: `pip install "avtomatika-worker[pydantic]"`.
266
+
267
+ Define a `pydantic.BaseModel` for your parameters. The SDK will automatically validate the incoming data against this model. If validation fails, the detailed error message from Pydantic will be sent to the orchestrator.
268
+
269
+ ```python
270
+ from pydantic import BaseModel, Field
271
+
272
+ class ResizeParams(BaseModel):
273
+ width: int = Field(gt=0, description="Width must be positive")
274
+ height: int = Field(gt=0, description="Height must be positive")
275
+ source_url: str
276
+
277
+ @worker.task("resize_image")
278
+ async def image_resizer(params: ResizeParams, **kwargs):
279
+ # Data is guaranteed to be valid
280
+ print(f"Resizing {params.source_url} to {params.width}x{params.height}")
281
+ # ...
282
+ ```
283
+
284
+ ## Key Features
285
+
286
+ ### 1. Task Handlers
287
+
288
+ Each handler is an asynchronous function that accepts two arguments:
289
+
290
+ - `params` (`dict`, `dataclass`, or `pydantic.BaseModel`): The parameters for the task, automatically validated and instantiated based on your type hint.
291
+ - `**kwargs`: Additional metadata about the task, including:
292
+ - `task_id` (`str`): The unique ID of the task itself.
293
+ - `job_id` (`str`): The ID of the parent `Job` to which the task belongs.
294
+ - `priority` (`int`): The execution priority of the task.
295
+
296
+ ### 2. Concurrency Limiting
297
+
298
+ The worker allows you to control how many tasks are executed in parallel. This can be configured at two levels:
299
+
300
+ - **Global Limit**: A maximum number of tasks that the worker can execute simultaneously, regardless of their type. This can be set with the `MAX_CONCURRENT_TASKS` environment variable or by passing `max_concurrent_tasks` to the `Worker` constructor.
301
+ - **Per-Type Limit**: A specific limit for a group of tasks that share a common resource (e.g., a GPU, a specific API), passed via `task_type_limits` to the `Worker` constructor.
302
+
303
+ The worker dynamically reports its available capacity to the orchestrator. When a limit is reached, the worker informs the orchestrator that it can no longer accept tasks of that type until a slot becomes free.
304
+
305
+ **Example:**
306
+
307
+ Let's configure a worker that can run up to **10 tasks in total**, but no more than **1 video processing task** and **4 audio transcription tasks** at the same time.
308
+
309
+ ```python
310
+ import asyncio
311
+ from avtomatika_worker import Worker
312
+
313
+ # 1. Configure limits during initialization
314
+ worker = Worker(
315
+ worker_type="media-processor",
316
+ max_concurrent_tasks=10,
317
+ task_type_limits={
318
+ "video_processing": 1,
319
+ "audio_processing": 4,
320
+ }
321
+ )
322
+
323
+ # 2. Assign a type to each task using the decorator
324
+ @worker.task("upscale_video", task_type="video_processing")
325
+ async def upscale_video(params: dict, **kwargs):
326
+ # This task uses the 'video_processing' slot
327
+ print("Upscaling video...")
328
+ await asyncio.sleep(5)
329
+ return {"status": "success"}
330
+
331
+ @worker.task("blur_video_faces", task_type="video_processing")
332
+ async def blur_video_faces(params: dict, **kwargs):
333
+ # This task also uses the 'video_processing' slot
334
+ print("Blurring faces in video...")
335
+ await asyncio.sleep(5)
336
+ return {"status": "success"}
337
+
338
+ @worker.task("transcribe_audio", task_type="audio_processing")
339
+ async def transcribe_audio(params: dict, **kwargs):
340
+ # This task uses one of the four 'audio_processing' slots
341
+ print("Transcribing audio...")
342
+ await asyncio.sleep(2)
343
+ return {"status": "success"}
344
+
345
+ @worker.task("generate_report")
346
+ async def generate_report(params: dict, **kwargs):
347
+ # This task has no specific type and is only limited by the global limit
348
+ print("Generating report...")
349
+ await asyncio.sleep(1)
350
+ return {"status": "success"}
351
+
352
+
353
+ if __name__ == "__main__":
354
+ worker.run_with_health_check()
355
+ ```
356
+ In this example, even though the global limit is 10, the orchestrator will only ever send one task (`upscale_video` or `blur_video_faces`) to this worker at a time, because they both share the single "video_processing" slot.
357
+
358
+ ### 3. Returning Results and Handling Errors
359
+
360
+ The result returned by a handler directly influences the subsequent flow of the pipeline in the orchestrator.
361
+
362
+ #### Successful Execution
363
+
364
+ ```python
365
+ return {
366
+ "status": "success",
367
+ "data": {"output": "some_value"}
368
+ }
369
+ ```
370
+ - The orchestrator will receive this data and use the `"success"` key in the `transitions` dictionary to determine the next step.
371
+
372
+ #### Custom Statuses
373
+
374
+ You can return custom statuses to implement complex branching logic in the orchestrator.
375
+ ```python
376
+ return {
377
+ "status": "needs_manual_review",
378
+ "data": {"reason": "Low confidence score"}
379
+ }
380
+ ```
381
+ - The orchestrator will look for the `"needs_manual_review"` key in `transitions`.
382
+
383
+ #### Error Handling
384
+
385
+ To control the orchestrator's fault tolerance mechanism, you can return standardized error types.
386
+
387
+ - **Transient Error (`TRANSIENT_ERROR`)**: For issues that might be resolved on a retry (e.g., a network failure).
388
+ ```python
389
+ from avtomatika_worker.typing import TRANSIENT_ERROR
390
+ return {
391
+ "status": "failure",
392
+ "error": {
393
+ "code": TRANSIENT_ERROR,
394
+ "message": "External API timeout"
395
+ }
396
+ }
397
+ ```
398
+ - **Permanent Error (`PERMANENT_ERROR`)**: For unresolvable problems (e.g., an invalid file format).
399
+ ```python
400
+ from avtomatika_worker.typing import PERMANENT_ERROR
401
+ return {
402
+ "status": "failure",
403
+ "error": {
404
+ "code": PERMANENT_ERROR,
405
+ "message": "Corrupted input file"
406
+ }
407
+ }
408
+ ```
409
+
410
+ ### 4. Failover and Load Balancing
411
+
412
+ The SDK supports connecting to multiple orchestrator instances to ensure high availability (`FAILOVER`) and load balancing (`ROUND_ROBIN`). This is configured via the `MULTI_ORCHESTRATOR_MODE` and `ORCHESTRATORS_CONFIG` environment variables.
413
+
414
+ **If `ORCHESTRATORS_CONFIG` is not set or is invalid JSON, the SDK will fall back to using the `ORCHESTRATOR_URL`. If neither is set, it will default to a single orchestrator at `http://localhost:8080`.** If both `ORCHESTRATORS_CONFIG` and `ORCHESTRATOR_URL` are set, `ORCHESTRATORS_CONFIG` will be used.
415
+
416
+ The `ORCHESTRATORS_CONFIG` variable must contain a JSON string. Each object in the list represents one orchestrator and can have the following keys:
417
+ - `url` (required): The URL of the orchestrator.
418
+ - `priority` (optional, default: 10): Used in `FAILOVER` mode. A lower number means a higher priority.
419
+ - `weight` (optional, default: 1): Used in `ROUND_ROBIN` mode to determine how frequently the orchestrator is polled.
420
+ - `token` (optional): A specific authentication token for this orchestrator. If not provided, the global `WORKER_TOKEN` is used.
421
+
422
+ **Example `ORCHESTRATORS_CONFIG`:**
423
+ ```json
424
+ [
425
+ {"url": "http://customer-a.com", "priority": 10, "weight": 100, "token": "token-for-customer-a"},
426
+ {"url": "http://customer-b.com", "priority": 10, "weight": 50, "token": "token-for-customer-b"},
427
+ {"url": "http://internal-backup.com", "priority": 20, "weight": 10}
428
+ ]
429
+ ```
430
+
431
+ - **`FAILOVER` (default):** The worker connects to orchestrators in the order of their `priority`. It will always try the highest-priority orchestrator first and only switch to the next one if the current one becomes unavailable. In the example above, it would try both `customer-a.com` and `customer-b.com` (which have the same priority) before trying `internal-backup.com`.
432
+ - **`ROUND_ROBIN`:** The worker distributes its requests to fetch tasks across all configured orchestrators based on their `weight`. An orchestrator with a higher weight will be polled for tasks more frequently. In the example, `customer-a.com` would be polled twice as often as `customer-b.com`.
433
+
434
+
435
+
436
+
437
+
438
+ ### 5. File System Helper (TaskFiles)
439
+
440
+ To simplify working with temporary files and paths, the SDK provides a `TaskFiles` helper class. It automatically manages directory creation within the isolated task folder and provides an asynchronous interface for file operations. Just add an argument typed as `TaskFiles` to your handler:
441
+
442
+ ```python
443
+ from avtomatika_worker import Worker, TaskFiles
444
+
445
+ @worker.task("generate_report")
446
+ async def generate_report(params: dict, files: TaskFiles, **kwargs):
447
+ # 1. Easy read/write
448
+ await files.write("data.json", '{"status": "ok"}')
449
+ content = await files.read("data.json")
450
+
451
+ # 2. Get path (directory is created automatically)
452
+ output_path = await files.path_to("report.pdf")
453
+
454
+ # 3. Check and list files
455
+ if await files.exists("input.jpg"):
456
+ file_list = await files.list()
457
+
458
+ return {"data": {"report": output_path}}
459
+ ```
460
+
461
+ **Available Methods (all asynchronous):**
462
+ - `await path_to(name)` — returns the full path to a file (ensures the task directory exists).
463
+ - `await read(name, mode='r')` — reads the entire file.
464
+ - `await write(name, data, mode='w')` — writes data to a file.
465
+ - `await list()` — lists filenames in the task directory.
466
+ - `await exists(name)` — checks if a file exists.
467
+ - `async with open(name, mode)` — async context manager for advanced usage.
468
+
469
+ > **Note: Automatic Cleanup**
470
+ >
471
+ > The SDK automatically deletes the entire task directory (including everything created via `TaskFiles`) immediately after the task completes and the result is sent.
472
+
473
+ ### 6. Handling Large Files (S3 Payload Offloading)
474
+
475
+ The SDK supports working with large files "out of the box" via S3-compatible storage.
476
+
477
+ - **Automatic Download**: If a value in `params` is a URI of the form `s3://...`, the SDK will automatically download the file to the local disk and replace the URI in `params` with the local path. **If the URI ends with `/` (e.g., `s3://bucket/data/`), the SDK treats it as a folder prefix and recursively downloads all matching objects into a local directory.**
478
+ - **Automatic Upload**: If your handler returns a local file path in `data` (located within the `TASK_FILES_DIR` directory), the SDK will automatically upload this file to S3 and replace the path with an `s3://` URI in the final result. **If the path is a directory, the SDK recursively uploads all files within it.**
479
+
480
+ This functionality is transparent to your code.
481
+
482
+ #### S3 Example
483
+
484
+ Suppose the orchestrator sends a task with `{"input_image": "s3://my-bucket/photo.jpg"}`:
485
+
486
+ ```python
487
+ import os
488
+ from avtomatika_worker import Worker, TaskFiles
489
+
490
+ worker = Worker(worker_type="image-worker")
491
+
492
+ @worker.task("process_image")
493
+ async def handle_image(params: dict, files: TaskFiles, **kwargs):
494
+ # SDK has already downloaded the file.
495
+ # 'input_image' now contains a local path like '/tmp/payloads/task-id/photo.jpg'
496
+ local_input = params["input_image"]
497
+ local_output = await files.path_to("processed.png")
498
+
499
+ # Your logic here (using local files)
500
+ # ... image processing ...
501
+
502
+ # Return the local path of the result.
503
+ # The SDK will upload it back to S3 automatically.
504
+ return {
505
+ "status": "success",
506
+ "data": {
507
+ "output_image": local_output
508
+ }
509
+ }
510
+ ```
511
+
512
+ This only requires configuring environment variables for S3 access (see Full Configuration Reference).
513
+
514
+ > **Important: S3 Consistency**
515
+ >
516
+ > The SDK **does not validate** that the Worker and Orchestrator share the same storage backend. You must ensure that:
517
+ > 1. The Worker can reach the `S3_ENDPOINT_URL` used by the Orchestrator.
518
+ > 2. The Worker's credentials allow reading from the buckets referenced in the incoming `s3://` URIs.
519
+ > 3. The Worker's credentials allow writing to the `S3_DEFAULT_BUCKET`.
520
+
521
+ ### 7. WebSocket Support
522
+
523
+ ## Advanced Features
524
+
525
+ ### Reporting Skill & Model Dependencies
526
+
527
+ For more advanced scheduling, the worker can report detailed information about its skills and their dependencies on specific models. This allows the orchestrator to make smarter decisions, such as dispatching tasks to workers that already have the required models loaded in memory.
528
+
529
+ This is configured via the `skill_dependencies` argument in the `Worker` constructor.
530
+
531
+ - **`skill_dependencies`**: A dictionary where keys are skill names (as registered with `@worker.task`) and values are.
532
+ The user wants to improve the `README.md` file. I've already read it and have a plan. I need to get the file content and then I can use the `replace` tool to update it.
533
+ I've already read the file content in the previous step. Now I will use the `replace` tool to update the file.
534
+ I have read the `README.md` file. Now I will reorder its sections to improve clarity for new users. The new order will be: Installation, Configuration, Quick Start, Key Features, Advanced Features, Full Configuration Reference, and Development.
535
+ I have read the `README.md` file. Now I will update it to document the new flexible parameter typing feature. I will add a new section called "Defining Task Parameters" and update the "Installation" section. lists of model names required by that skill.
536
+
537
+ Based on this configuration and the current state of the worker's `hot_cache` (the set of models currently loaded in memory), the worker will automatically include two new fields in its heartbeat messages:
538
+
539
+ - **`skill_dependencies`**: The same dictionary provided during initialization.
540
+ - **`hot_skills`**: A dynamically calculated list of skills that are ready for immediate execution (i.e., all of their dependent models are in the `hot_cache`).
541
+
542
+ **Example:**
543
+
544
+ Consider a worker configured like this:
545
+ ```python
546
+ worker = Worker(
547
+ worker_type="ai-processor",
548
+ skill_dependencies={
549
+ "image_generation": ["stable_diffusion_v1.5", "vae-ft-mse"],
550
+ "upscale": ["realesrgan_x4"],
551
+ }
552
+ )
553
+ ```
554
+
555
+ - Initially, `hot_cache` is empty. The worker's heartbeat will include `skill_dependencies` but not `hot_skills`.
556
+ - A task handler calls `add_to_hot_cache("stable_diffusion_v1.5")`. The next heartbeat will still not include `hot_skills` because the `image_generation` skill is only partially loaded.
557
+ - The handler then calls `add_to_hot_cache("vae-ft-mse")`. Now, all dependencies for `image_generation` are met. The next heartbeat will include:
558
+ ```json
559
+ {
560
+ "hot_skills": ["image_generation"],
561
+ "skill_dependencies": {
562
+ "image_generation": ["stable_diffusion_v1.5", "vae-ft-mse"],
563
+ "upscale": ["realesrgan_x4"]
564
+ }
565
+ }
566
+ ```
567
+ This information is sent automatically. Your task handlers are only responsible for managing the `hot_cache` by calling `add_to_hot_cache()` and `remove_from_hot_cache()`, which are passed as arguments to the handler.
568
+
569
+ ## Full Configuration Reference
570
+
571
+ The worker is fully configured via environment variables.
572
+
573
+ | Variable | Description | Default |
574
+ | ----------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------- |
575
+ | `WORKER_ID` | A unique identifier for the worker. | A random UUID, e.g., `worker-...` |
576
+ | `WORKER_TYPE` | A string identifying the type of the worker. | `generic-cpu-worker` |
577
+ | `WORKER_PORT` | The port for the worker's health check server. | `8083` |
578
+ | `WORKER_TOKEN` | A common authentication token used to connect to orchestrators. | `your-secret-worker-token` |
579
+ | `WORKER_INDIVIDUAL_TOKEN` | An individual token for this worker, which overrides `WORKER_TOKEN` if set. | - |
580
+ | `ORCHESTRATOR_URL` | The URL of a single orchestrator (used if `ORCHESTRATORS_CONFIG` is not set). | `http://localhost:8080` |
581
+ | `ORCHESTRATORS_CONFIG` | A JSON string with a list of orchestrators for multi-orchestrator modes. | `[]` |
582
+ | `MULTI_ORCHESTRATOR_MODE` | The mode for handling multiple orchestrators. Possible values: `FAILOVER`, `ROUND_ROBIN`. | `FAILOVER` |
583
+ | `MAX_CONCURRENT_TASKS` | The maximum number of tasks the worker can execute simultaneously. | `10` |
584
+ | `COST_PER_SKILL` | A JSON string mapping skill names to their cost per second. | `{}` |
585
+ | `CPU_CORES` | The number of CPU cores available to the worker. | `4` |
586
+ | `GPU_MODEL` | The model of the GPU available to the worker (e.g., "RTX 4090"). | - |
587
+ | `GPU_VRAM_GB` | The amount of VRAM in GB for the GPU. | `0` |
588
+ | `INSTALLED_SOFTWARE` | A JSON string representing a dictionary of installed software and their versions. | `{"python": "3.9"}` |
589
+ | `INSTALLED_MODELS` | A JSON string representing a list of dictionaries with information about installed models. | `[]` |
590
+ | `HEARTBEAT_INTERVAL` | The interval in seconds between heartbeats to the orchestrator. | `15` |
591
+ | `WORKER_HEARTBEAT_DEBOUNCE_DELAY` | The delay in seconds for debouncing immediate heartbeats after a state change. | `0.1` |
592
+ | `WORKER_ENABLE_WEBSOCKETS` | Enable (`true`) or disable (`false`) WebSocket support for real-time commands. | `false` |
593
+ | `RESULT_MAX_RETRIES` | The maximum number of times to retry sending a task result if it fails. | `5` |
594
+ | `RESULT_RETRY_INITIAL_DELAY` | The initial delay in seconds before the first retry of sending a result. | `1.0` |
595
+ | `TASK_POLL_TIMEOUT` | The timeout in seconds for polling for new tasks. | `30` |
596
+ | `TASK_POLL_ERROR_DELAY` | The delay in seconds before retrying after a polling error. | `5.0` |
597
+ | `IDLE_POLL_DELAY` | The delay in seconds between polls when the worker is idle. | `0.01` |
598
+ | `TASK_FILES_DIR` | The directory for temporarily storing files when working with S3. | `/tmp/payloads` |
599
+ | `S3_ENDPOINT_URL` | The URL of the S3-compatible storage. | - |
600
+ | `S3_ACCESS_KEY` | The access key for S3. | - |
601
+ | `S3_SECRET_KEY` | The secret key for S3. | - |
602
+ | `S3_DEFAULT_BUCKET` | The default bucket name for uploading results. | `avtomatika-payloads` |
603
+
604
+ ## Development
605
+
606
+ To install the necessary dependencies for running tests, use the following command:
607
+
608
+ ```bash
609
+ pip install .[test]
610
+ ```