pyworkflow-engine 0.1.7__py3-none-any.whl → 0.1.10__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (146) hide show
  1. pyworkflow/__init__.py +10 -1
  2. pyworkflow/celery/tasks.py +272 -24
  3. pyworkflow/cli/__init__.py +4 -1
  4. pyworkflow/cli/commands/runs.py +4 -4
  5. pyworkflow/cli/commands/setup.py +203 -4
  6. pyworkflow/cli/utils/config_generator.py +76 -3
  7. pyworkflow/cli/utils/docker_manager.py +232 -0
  8. pyworkflow/config.py +94 -17
  9. pyworkflow/context/__init__.py +13 -0
  10. pyworkflow/context/base.py +26 -0
  11. pyworkflow/context/local.py +80 -0
  12. pyworkflow/context/step_context.py +295 -0
  13. pyworkflow/core/registry.py +6 -1
  14. pyworkflow/core/step.py +141 -0
  15. pyworkflow/core/workflow.py +56 -0
  16. pyworkflow/engine/events.py +30 -0
  17. pyworkflow/engine/replay.py +39 -0
  18. pyworkflow/primitives/child_workflow.py +1 -1
  19. pyworkflow/runtime/local.py +1 -1
  20. pyworkflow/storage/__init__.py +14 -0
  21. pyworkflow/storage/base.py +35 -0
  22. pyworkflow/storage/cassandra.py +1747 -0
  23. pyworkflow/storage/config.py +69 -0
  24. pyworkflow/storage/dynamodb.py +31 -2
  25. pyworkflow/storage/file.py +28 -0
  26. pyworkflow/storage/memory.py +18 -0
  27. pyworkflow/storage/mysql.py +1159 -0
  28. pyworkflow/storage/postgres.py +27 -2
  29. pyworkflow/storage/schemas.py +4 -3
  30. pyworkflow/storage/sqlite.py +25 -2
  31. {pyworkflow_engine-0.1.7.dist-info → pyworkflow_engine-0.1.10.dist-info}/METADATA +7 -4
  32. pyworkflow_engine-0.1.10.dist-info/RECORD +91 -0
  33. pyworkflow_engine-0.1.10.dist-info/top_level.txt +1 -0
  34. dashboard/backend/app/__init__.py +0 -1
  35. dashboard/backend/app/config.py +0 -32
  36. dashboard/backend/app/controllers/__init__.py +0 -6
  37. dashboard/backend/app/controllers/run_controller.py +0 -86
  38. dashboard/backend/app/controllers/workflow_controller.py +0 -33
  39. dashboard/backend/app/dependencies/__init__.py +0 -5
  40. dashboard/backend/app/dependencies/storage.py +0 -50
  41. dashboard/backend/app/repositories/__init__.py +0 -6
  42. dashboard/backend/app/repositories/run_repository.py +0 -80
  43. dashboard/backend/app/repositories/workflow_repository.py +0 -27
  44. dashboard/backend/app/rest/__init__.py +0 -8
  45. dashboard/backend/app/rest/v1/__init__.py +0 -12
  46. dashboard/backend/app/rest/v1/health.py +0 -33
  47. dashboard/backend/app/rest/v1/runs.py +0 -133
  48. dashboard/backend/app/rest/v1/workflows.py +0 -41
  49. dashboard/backend/app/schemas/__init__.py +0 -23
  50. dashboard/backend/app/schemas/common.py +0 -16
  51. dashboard/backend/app/schemas/event.py +0 -24
  52. dashboard/backend/app/schemas/hook.py +0 -25
  53. dashboard/backend/app/schemas/run.py +0 -54
  54. dashboard/backend/app/schemas/step.py +0 -28
  55. dashboard/backend/app/schemas/workflow.py +0 -31
  56. dashboard/backend/app/server.py +0 -87
  57. dashboard/backend/app/services/__init__.py +0 -6
  58. dashboard/backend/app/services/run_service.py +0 -240
  59. dashboard/backend/app/services/workflow_service.py +0 -155
  60. dashboard/backend/main.py +0 -18
  61. docs/concepts/cancellation.mdx +0 -362
  62. docs/concepts/continue-as-new.mdx +0 -434
  63. docs/concepts/events.mdx +0 -266
  64. docs/concepts/fault-tolerance.mdx +0 -370
  65. docs/concepts/hooks.mdx +0 -552
  66. docs/concepts/limitations.mdx +0 -167
  67. docs/concepts/schedules.mdx +0 -775
  68. docs/concepts/sleep.mdx +0 -312
  69. docs/concepts/steps.mdx +0 -301
  70. docs/concepts/workflows.mdx +0 -255
  71. docs/guides/cli.mdx +0 -942
  72. docs/guides/configuration.mdx +0 -560
  73. docs/introduction.mdx +0 -155
  74. docs/quickstart.mdx +0 -279
  75. examples/__init__.py +0 -1
  76. examples/celery/__init__.py +0 -1
  77. examples/celery/durable/docker-compose.yml +0 -55
  78. examples/celery/durable/pyworkflow.config.yaml +0 -12
  79. examples/celery/durable/workflows/__init__.py +0 -122
  80. examples/celery/durable/workflows/basic.py +0 -87
  81. examples/celery/durable/workflows/batch_processing.py +0 -102
  82. examples/celery/durable/workflows/cancellation.py +0 -273
  83. examples/celery/durable/workflows/child_workflow_patterns.py +0 -240
  84. examples/celery/durable/workflows/child_workflows.py +0 -202
  85. examples/celery/durable/workflows/continue_as_new.py +0 -260
  86. examples/celery/durable/workflows/fault_tolerance.py +0 -210
  87. examples/celery/durable/workflows/hooks.py +0 -211
  88. examples/celery/durable/workflows/idempotency.py +0 -112
  89. examples/celery/durable/workflows/long_running.py +0 -99
  90. examples/celery/durable/workflows/retries.py +0 -101
  91. examples/celery/durable/workflows/schedules.py +0 -209
  92. examples/celery/transient/01_basic_workflow.py +0 -91
  93. examples/celery/transient/02_fault_tolerance.py +0 -257
  94. examples/celery/transient/__init__.py +0 -20
  95. examples/celery/transient/pyworkflow.config.yaml +0 -25
  96. examples/local/__init__.py +0 -1
  97. examples/local/durable/01_basic_workflow.py +0 -94
  98. examples/local/durable/02_file_storage.py +0 -132
  99. examples/local/durable/03_retries.py +0 -169
  100. examples/local/durable/04_long_running.py +0 -119
  101. examples/local/durable/05_event_log.py +0 -145
  102. examples/local/durable/06_idempotency.py +0 -148
  103. examples/local/durable/07_hooks.py +0 -334
  104. examples/local/durable/08_cancellation.py +0 -233
  105. examples/local/durable/09_child_workflows.py +0 -198
  106. examples/local/durable/10_child_workflow_patterns.py +0 -265
  107. examples/local/durable/11_continue_as_new.py +0 -249
  108. examples/local/durable/12_schedules.py +0 -198
  109. examples/local/durable/__init__.py +0 -1
  110. examples/local/transient/01_quick_tasks.py +0 -87
  111. examples/local/transient/02_retries.py +0 -130
  112. examples/local/transient/03_sleep.py +0 -141
  113. examples/local/transient/__init__.py +0 -1
  114. pyworkflow_engine-0.1.7.dist-info/RECORD +0 -196
  115. pyworkflow_engine-0.1.7.dist-info/top_level.txt +0 -5
  116. tests/examples/__init__.py +0 -0
  117. tests/integration/__init__.py +0 -0
  118. tests/integration/test_cancellation.py +0 -330
  119. tests/integration/test_child_workflows.py +0 -439
  120. tests/integration/test_continue_as_new.py +0 -428
  121. tests/integration/test_dynamodb_storage.py +0 -1146
  122. tests/integration/test_fault_tolerance.py +0 -369
  123. tests/integration/test_schedule_storage.py +0 -484
  124. tests/unit/__init__.py +0 -0
  125. tests/unit/backends/__init__.py +0 -1
  126. tests/unit/backends/test_dynamodb_storage.py +0 -1554
  127. tests/unit/backends/test_postgres_storage.py +0 -1281
  128. tests/unit/backends/test_sqlite_storage.py +0 -1460
  129. tests/unit/conftest.py +0 -41
  130. tests/unit/test_cancellation.py +0 -364
  131. tests/unit/test_child_workflows.py +0 -680
  132. tests/unit/test_continue_as_new.py +0 -441
  133. tests/unit/test_event_limits.py +0 -316
  134. tests/unit/test_executor.py +0 -320
  135. tests/unit/test_fault_tolerance.py +0 -334
  136. tests/unit/test_hooks.py +0 -495
  137. tests/unit/test_registry.py +0 -261
  138. tests/unit/test_replay.py +0 -420
  139. tests/unit/test_schedule_schemas.py +0 -285
  140. tests/unit/test_schedule_utils.py +0 -286
  141. tests/unit/test_scheduled_workflow.py +0 -274
  142. tests/unit/test_step.py +0 -353
  143. tests/unit/test_workflow.py +0 -243
  144. {pyworkflow_engine-0.1.7.dist-info → pyworkflow_engine-0.1.10.dist-info}/WHEEL +0 -0
  145. {pyworkflow_engine-0.1.7.dist-info → pyworkflow_engine-0.1.10.dist-info}/entry_points.txt +0 -0
  146. {pyworkflow_engine-0.1.7.dist-info → pyworkflow_engine-0.1.10.dist-info}/licenses/LICENSE +0 -0
docs/guides/cli.mdx DELETED
@@ -1,942 +0,0 @@
1
- ---
2
- title: 'Command Line Interface'
3
- description: 'Manage workflows and runs from the terminal with the PyWorkflow CLI'
4
- ---
5
-
6
- ## Overview
7
-
8
- PyWorkflow includes a powerful CLI for managing workflows and monitoring runs directly from your terminal. The CLI provides commands to list, inspect, and execute workflows, as well as monitor their execution status and event logs.
9
-
10
- <CardGroup cols={2}>
11
- <Card title="Quickstart" icon="rocket">
12
- Create a new project with sample workflows in seconds.
13
- </Card>
14
- <Card title="Workflow Management" icon="diagram-project">
15
- List, inspect, and run workflows from the command line.
16
- </Card>
17
- <Card title="Run Monitoring" icon="chart-line">
18
- Check run status, view event logs, and debug executions.
19
- </Card>
20
- <Card title="Schedule Management" icon="calendar">
21
- Create, manage, and monitor automated workflow schedules.
22
- </Card>
23
- <Card title="Scheduler" icon="clock">
24
- Run the local scheduler for triggering due schedules.
25
- </Card>
26
- <Card title="Worker Management" icon="server">
27
- Start and manage Celery workers for distributed execution.
28
- </Card>
29
- </CardGroup>
30
-
31
- ## Installation
32
-
33
- The CLI is included with PyWorkflow and available as the `pyworkflow` command:
34
-
35
- ```bash
36
- pip install pyworkflow
37
- pyworkflow --version
38
- ```
39
-
40
- ## Global Options
41
-
42
- These options apply to all commands:
43
-
44
- | Option | Environment Variable | Description |
45
- |--------|---------------------|-------------|
46
- | `--module` | `PYWORKFLOW_MODULE` | Python module to import for workflow discovery |
47
- | `--runtime` | `PYWORKFLOW_RUNTIME` | Execution runtime: `local` (in-process) or `celery` (distributed). Default: celery |
48
- | `--storage` | `PYWORKFLOW_STORAGE_BACKEND` | Storage backend: `file` or `memory` (default: file) |
49
- | `--storage-path` | `PYWORKFLOW_STORAGE_PATH` | Path for file storage (default: ./workflow_data) |
50
- | `--output` | - | Output format: `table`, `json`, or `plain` (default: table) |
51
- | `--verbose`, `-v` | - | Enable verbose logging |
52
- | `--version` | - | Show version information |
53
-
54
- ## Configuration
55
-
56
- ### Configuration File
57
-
58
- Create a `pyworkflow.config.yaml` file in your project directory:
59
-
60
- ```yaml
61
- # pyworkflow.config.yaml
62
- module: myapp.workflows
63
-
64
- runtime: celery # or "local"
65
-
66
- storage:
67
- backend: file
68
- path: ./workflow_data
69
-
70
- celery:
71
- broker: redis://localhost:6379/0
72
- result_backend: redis://localhost:6379/1
73
- ```
74
-
75
- <Note>
76
- The YAML config file is the **recommended** approach. Place it in your working directory
77
- and both the CLI and your Python code will automatically use it.
78
- </Note>
79
-
80
- ### Alternative Config Formats
81
-
82
- PyWorkflow also supports TOML configuration files (searched in order, walking up the directory tree):
83
-
84
- 1. `pyworkflow.toml`
85
- 2. `.pyworkflow.toml`
86
- 3. `pyproject.toml` (under `[tool.pyworkflow]` section)
87
-
88
- <Accordion title="TOML Configuration Examples">
89
- <Tabs>
90
- <Tab title="pyworkflow.toml">
91
- ```toml
92
- module = "myapp.workflows"
93
- runtime = "celery"
94
-
95
- [storage]
96
- backend = "file"
97
- path = "./workflow_data"
98
-
99
- [celery]
100
- broker = "redis://localhost:6379/0"
101
- result_backend = "redis://localhost:6379/1"
102
- ```
103
- </Tab>
104
- <Tab title="pyproject.toml">
105
- ```toml
106
- [tool.pyworkflow]
107
- module = "myapp.workflows"
108
- runtime = "celery"
109
-
110
- [tool.pyworkflow.storage]
111
- backend = "file"
112
- path = "./workflow_data"
113
-
114
- [tool.pyworkflow.celery]
115
- broker = "redis://localhost:6379/0"
116
- result_backend = "redis://localhost:6379/1"
117
- ```
118
- </Tab>
119
- </Tabs>
120
- </Accordion>
121
-
122
- ### Priority Resolution
123
-
124
- Configuration values are resolved in this order (highest to lowest priority):
125
-
126
- | Priority | Source | Example |
127
- |----------|--------|---------|
128
- | 1 (highest) | CLI flags | `--module myapp.workflows` |
129
- | 2 | Environment variables | `PYWORKFLOW_MODULE=myapp.workflows` |
130
- | 3 | Config file | `pyworkflow.config.yaml` |
131
- | 4 (lowest) | Defaults | `runtime: local`, `durable: false` |
132
-
133
- ---
134
-
135
- ## Workflow Discovery
136
-
137
- When you run `pyworkflow worker run` or other CLI commands, PyWorkflow needs to discover
138
- and import your workflow modules. This happens in the following priority order:
139
-
140
- ### Discovery Priority
141
-
142
- | Priority | Source | Example |
143
- |----------|--------|---------|
144
- | 1 (highest) | `--module` flag | `pyworkflow --module myapp.workflows worker run` |
145
- | 2 | `PYWORKFLOW_DISCOVER` env var | `PYWORKFLOW_DISCOVER=myapp.workflows pyworkflow worker run` |
146
- | 3 (lowest) | `pyworkflow.config.yaml` | `module: myapp.workflows` in config file |
147
-
148
- ### How Discovery Works
149
-
150
- 1. **Module Import**: PyWorkflow imports the specified Python module(s)
151
- 2. **Decorator Registration**: When the module loads, `@workflow` and `@step` decorators
152
- automatically register functions in the global registry
153
- 3. **Project Root Detection**: PyWorkflow automatically finds your project root (by looking
154
- for `pyproject.toml`, `setup.py`, or `.git`) and adds it to the Python path
155
-
156
- <Tabs>
157
- <Tab title="Using Config File (Recommended)">
158
- ```bash
159
- # Create pyworkflow.config.yaml in your project
160
- cd myproject/
161
- cat > pyworkflow.config.yaml << EOF
162
- module: myapp.workflows
163
- runtime: celery
164
- storage:
165
- backend: file
166
- path: ./workflow_data
167
- celery:
168
- broker: redis://localhost:6379/0
169
- EOF
170
-
171
- # Now just run - config is auto-detected
172
- pyworkflow worker run
173
- ```
174
- </Tab>
175
- <Tab title="Using --module Flag">
176
- ```bash
177
- pyworkflow --module myapp.workflows worker run
178
- ```
179
- </Tab>
180
- <Tab title="Using Environment Variable">
181
- ```bash
182
- export PYWORKFLOW_DISCOVER=myapp.workflows
183
- pyworkflow worker run
184
-
185
- # Or inline
186
- PYWORKFLOW_DISCOVER=myapp.workflows pyworkflow worker run
187
- ```
188
- </Tab>
189
- </Tabs>
190
-
191
- ### Multiple Modules
192
-
193
- You can discover workflows from multiple modules:
194
-
195
- <Tabs>
196
- <Tab title="Config File">
197
- ```yaml
198
- # pyworkflow.config.yaml
199
- modules:
200
- - myapp.workflows
201
- - myapp.tasks
202
- - myapp.handlers
203
- ```
204
- </Tab>
205
- <Tab title="Environment Variable">
206
- ```bash
207
- # Comma-separated list
208
- PYWORKFLOW_DISCOVER=myapp.workflows,myapp.tasks pyworkflow worker run
209
- ```
210
- </Tab>
211
- </Tabs>
212
-
213
- ## Commands
214
-
215
- ### Workflow Commands
216
-
217
- Manage and execute registered workflows.
218
-
219
- #### `workflows list`
220
-
221
- List all registered workflows:
222
-
223
- ```bash
224
- pyworkflow --module myapp.workflows workflows list
225
- ```
226
-
227
- **Output Formats:**
228
- - `table` (default): Shows Name, Max Duration, and Metadata columns
229
- - `json`: Array of workflow objects
230
- - `plain`: Simple list of workflow names
231
-
232
- #### `workflows info`
233
-
234
- Show detailed information about a specific workflow:
235
-
236
- ```bash
237
- pyworkflow --module myapp.workflows workflows info my_workflow
238
- ```
239
-
240
- **Arguments:**
241
- | Argument | Required | Description |
242
- |----------|----------|-------------|
243
- | `WORKFLOW_NAME` | Yes | Name of the workflow to inspect |
244
-
245
- **Output includes:** Name, max duration, function details, module path, and docstring.
246
-
247
- #### `workflows run`
248
-
249
- Execute a workflow with optional arguments:
250
-
251
- ```bash
252
- pyworkflow --module myapp.workflows workflows run my_workflow \
253
- --arg user_id=123 --arg amount=50.00
254
- ```
255
-
256
- **Arguments:**
257
- | Argument | Required | Description |
258
- |----------|----------|-------------|
259
- | `WORKFLOW_NAME` | Yes | Name of the workflow to run |
260
-
261
- **Options:**
262
- | Option | Description |
263
- |--------|-------------|
264
- | `--arg key=value` | Workflow argument (repeatable, supports JSON values) |
265
- | `--args-json '{...}'` | Workflow arguments as JSON object |
266
- | `--durable/--no-durable` | Run in durable mode (default: durable) |
267
- | `--idempotency-key` | Idempotency key for the execution |
268
-
269
- <Tabs>
270
- <Tab title="Key-Value Arguments">
271
- ```bash
272
- pyworkflow workflows run order_process \
273
- --arg order_id=12345 \
274
- --arg amount=99.99 \
275
- --arg items='["item1", "item2"]'
276
- ```
277
- </Tab>
278
- <Tab title="JSON Arguments">
279
- ```bash
280
- pyworkflow workflows run order_process \
281
- --args-json '{"order_id": 12345, "amount": 99.99, "items": ["item1", "item2"]}'
282
- ```
283
- </Tab>
284
- </Tabs>
285
-
286
- <Tip>
287
- Use `--no-durable` for quick, transient executions that don't need persistence. Use `--idempotency-key` to prevent duplicate executions.
288
- </Tip>
289
-
290
- ---
291
-
292
- ### Run Commands
293
-
294
- Monitor and debug workflow runs.
295
-
296
- #### `runs list`
297
-
298
- List workflow runs with optional filtering:
299
-
300
- ```bash
301
- pyworkflow runs list --workflow my_workflow --status completed --limit 10
302
- ```
303
-
304
- **Options:**
305
- | Option | Description |
306
- |--------|-------------|
307
- | `--workflow` | Filter by workflow name |
308
- | `--status` | Filter by status: `pending`, `running`, `suspended`, `completed`, `failed`, `cancelled` |
309
- | `--limit` | Maximum runs to display (default: 20) |
310
-
311
- **Output Formats:**
312
- - `table` (default): Shows Run ID, Workflow, Status (color-coded), Started time, and Duration
313
- - `json`: Array of run objects with full details
314
- - `plain`: Simple list of Run IDs
315
-
316
- #### `runs status`
317
-
318
- Show detailed status of a specific run:
319
-
320
- ```bash
321
- pyworkflow runs status run_abc123def456
322
- ```
323
-
324
- **Arguments:**
325
- | Argument | Required | Description |
326
- |----------|----------|-------------|
327
- | `RUN_ID` | Yes | Workflow run identifier |
328
-
329
- **Output includes:**
330
- - Run ID, Workflow name, Status
331
- - Created, Started, Completed timestamps
332
- - Duration
333
- - Input arguments
334
- - Result (if completed)
335
- - Error message (if failed)
336
-
337
- #### `runs logs`
338
-
339
- View the execution event log for a run:
340
-
341
- ```bash
342
- pyworkflow runs logs run_abc123 --filter step_completed
343
- ```
344
-
345
- **Arguments:**
346
- | Argument | Required | Description |
347
- |----------|----------|-------------|
348
- | `RUN_ID` | Yes | Workflow run identifier |
349
-
350
- **Options:**
351
- | Option | Description |
352
- |--------|-------------|
353
- | `--filter` | Filter events by type (case-insensitive substring match) |
354
-
355
- **Event Types:**
356
- - `workflow_started`, `workflow_completed`, `workflow_failed`, `workflow_cancelled`
357
- - `step_started`, `step_completed`, `step_failed`
358
- - `sleep_started`, `sleep_resumed`
359
- - `hook_created`, `hook_received`
360
- - `cancellation_requested`
361
-
362
- #### `runs cancel`
363
-
364
- Cancel a running or suspended workflow:
365
-
366
- ```bash
367
- pyworkflow runs cancel run_abc123 --reason "User requested"
368
- ```
369
-
370
- **Arguments:**
371
- | Argument | Required | Description |
372
- |----------|----------|-------------|
373
- | `RUN_ID` | Yes | Workflow run identifier |
374
-
375
- **Options:**
376
- | Option | Description |
377
- |--------|-------------|
378
- | `--wait/--no-wait` | Wait for cancellation to complete (default: no-wait) |
379
- | `--timeout` | Timeout in seconds when waiting (default: 30) |
380
- | `--reason` | Reason for cancellation |
381
-
382
- **Examples:**
383
-
384
- ```bash
385
- # Cancel a workflow
386
- pyworkflow runs cancel run_abc123
387
-
388
- # Cancel with reason
389
- pyworkflow runs cancel run_abc123 --reason "Customer cancelled order"
390
-
391
- # Wait for cancellation to complete
392
- pyworkflow runs cancel run_abc123 --wait --timeout 60
393
- ```
394
-
395
- <Note>
396
- Cancellation is graceful - the workflow will stop at the next checkpoint (before a step, sleep, or hook), not immediately. See [Cancellation](/concepts/cancellation) for details.
397
- </Note>
398
-
399
- ---
400
-
401
- ### Worker Commands
402
-
403
- Manage Celery workers for distributed workflow execution.
404
-
405
- #### `worker run`
406
-
407
- Start a Celery worker to process workflow tasks:
408
-
409
- ```bash
410
- pyworkflow worker run
411
- ```
412
-
413
- **Options:**
414
- | Option | Description |
415
- |--------|-------------|
416
- | `--workflow` | Only process workflow orchestration tasks |
417
- | `--step` | Only process step execution tasks |
418
- | `--schedule` | Only process scheduled resumption tasks |
419
- | `--concurrency N` | Number of worker processes (default: auto) |
420
- | `--loglevel LEVEL` | Log level: `debug`, `info`, `warning`, `error` |
421
- | `--hostname NAME` | Custom worker hostname |
422
- | `--beat` | Also start Celery Beat scheduler |
423
-
424
- <Tabs>
425
- <Tab title="All Queues (Default)">
426
- ```bash
427
- # Start a worker processing all queues
428
- pyworkflow worker run
429
- ```
430
- </Tab>
431
- <Tab title="Specialized Workers">
432
- ```bash
433
- # Terminal 1: Workflow orchestration
434
- pyworkflow worker run --workflow
435
-
436
- # Terminal 2: Step execution (scale this for heavy work)
437
- pyworkflow worker run --step --concurrency 4
438
-
439
- # Terminal 3: Scheduled tasks
440
- pyworkflow worker run --schedule
441
- ```
442
- </Tab>
443
- </Tabs>
444
-
445
- <Tip>
446
- For production, run separate workers for each queue type. Scale step workers horizontally for computation-heavy workloads.
447
- </Tip>
448
-
449
- #### `worker status`
450
-
451
- Show status of active Celery workers:
452
-
453
- ```bash
454
- pyworkflow worker status
455
- ```
456
-
457
- Displays worker names, status, concurrency, active tasks, and processed task counts.
458
-
459
- #### `worker queues`
460
-
461
- Show available task queues and their configuration:
462
-
463
- ```bash
464
- pyworkflow worker queues
465
- ```
466
-
467
- ---
468
-
469
- ### Scheduler Commands
470
-
471
- Run the schedule executor for local runtime.
472
-
473
- #### `scheduler run`
474
-
475
- Start the local scheduler that polls for due schedules:
476
-
477
- ```bash
478
- pyworkflow scheduler run
479
- ```
480
-
481
- **Options:**
482
- | Option | Description |
483
- |--------|-------------|
484
- | `--poll-interval` | Seconds between storage polls (default: 5.0) |
485
- | `--duration` | Run for specified seconds then exit (default: run forever) |
486
-
487
- <Tabs>
488
- <Tab title="Basic Usage">
489
- ```bash
490
- # Start scheduler with defaults
491
- pyworkflow scheduler run
492
-
493
- # With custom poll interval
494
- pyworkflow scheduler run --poll-interval 10
495
- ```
496
- </Tab>
497
- <Tab title="With Module">
498
- ```bash
499
- # Discover workflows from a module
500
- pyworkflow --module myapp.workflows scheduler run
501
- ```
502
- </Tab>
503
- <Tab title="Testing">
504
- ```bash
505
- # Run for 60 seconds (useful for testing)
506
- pyworkflow scheduler run --duration 60
507
- ```
508
- </Tab>
509
- </Tabs>
510
-
511
- <Note>
512
- Use `scheduler run` for local runtime. For Celery runtime, use `worker run --beat`
513
- or start Celery Beat separately.
514
- </Note>
515
-
516
- ---
517
-
518
- ### Schedule Commands
519
-
520
- Manage workflow schedules for automated execution.
521
-
522
- #### `schedules list`
523
-
524
- List all schedules with optional filtering:
525
-
526
- ```bash
527
- pyworkflow schedules list --workflow my_workflow --status active --limit 10
528
- ```
529
-
530
- **Options:**
531
- | Option | Description |
532
- |--------|-------------|
533
- | `--workflow` | Filter by workflow name |
534
- | `--status` | Filter by status: `active`, `paused`, `deleted` |
535
- | `--limit` | Maximum schedules to display (default: 20) |
536
-
537
- **Output includes:** Schedule ID, Workflow, Status, Schedule description, Next Run time, Success rate
538
-
539
- #### `schedules create`
540
-
541
- Create a new schedule for a workflow:
542
-
543
- ```bash
544
- pyworkflow schedules create my_workflow --cron "0 9 * * *"
545
- ```
546
-
547
- **Arguments:**
548
- | Argument | Required | Description |
549
- |----------|----------|-------------|
550
- | `WORKFLOW_NAME` | Yes | Name of the workflow to schedule |
551
-
552
- **Options:**
553
- | Option | Description |
554
- |--------|-------------|
555
- | `--cron` | Cron expression (e.g., `"0 9 * * *"` for daily at 9 AM) |
556
- | `--interval` | Interval duration (e.g., `5m`, `1h`, `30s`) |
557
- | `--timezone` | Timezone for schedule (default: UTC) |
558
- | `--overlap` | Overlap policy: `skip`, `buffer_one`, `buffer_all`, `cancel_other`, `allow_all` |
559
- | `--schedule-id` | Custom schedule ID (optional) |
560
-
561
- <Tabs>
562
- <Tab title="Cron Schedule">
563
- ```bash
564
- # Every day at 9 AM
565
- pyworkflow schedules create daily_report --cron "0 9 * * *"
566
-
567
- # Every Monday at midnight
568
- pyworkflow schedules create weekly_cleanup --cron "0 0 * * 1"
569
- ```
570
- </Tab>
571
- <Tab title="Interval Schedule">
572
- ```bash
573
- # Every 5 minutes
574
- pyworkflow schedules create health_check --interval 5m
575
-
576
- # Every hour
577
- pyworkflow schedules create sync_data --interval 1h
578
- ```
579
- </Tab>
580
- <Tab title="With Options">
581
- ```bash
582
- pyworkflow schedules create my_workflow \
583
- --cron "0 9 * * *" \
584
- --timezone "America/New_York" \
585
- --overlap buffer_one \
586
- --schedule-id my_custom_id
587
- ```
588
- </Tab>
589
- </Tabs>
590
-
591
- #### `schedules show`
592
-
593
- Show detailed information about a schedule:
594
-
595
- ```bash
596
- pyworkflow schedules show sched_abc123
597
- ```
598
-
599
- **Output includes:**
600
- - Schedule ID, Workflow name, Status
601
- - Schedule specification (cron/interval)
602
- - Overlap policy, Timezone
603
- - Next run time, Last run time
604
- - Statistics: Total runs, Successful, Failed, Skipped
605
-
606
- #### `schedules pause`
607
-
608
- Pause a schedule (stops triggering new runs):
609
-
610
- ```bash
611
- pyworkflow schedules pause sched_abc123
612
- ```
613
-
614
- #### `schedules resume`
615
-
616
- Resume a paused schedule:
617
-
618
- ```bash
619
- pyworkflow schedules resume sched_abc123
620
- ```
621
-
622
- **Output includes:** New next run time after resumption
623
-
624
- #### `schedules delete`
625
-
626
- Delete a schedule (soft delete):
627
-
628
- ```bash
629
- pyworkflow schedules delete sched_abc123
630
- pyworkflow schedules delete sched_abc123 --force # Skip confirmation
631
- ```
632
-
633
- **Options:**
634
- | Option | Description |
635
- |--------|-------------|
636
- | `--force` | Delete without confirmation prompt |
637
-
638
- #### `schedules trigger`
639
-
640
- Manually trigger a schedule immediately:
641
-
642
- ```bash
643
- pyworkflow schedules trigger sched_abc123
644
- ```
645
-
646
- This executes the workflow immediately without affecting the regular schedule timing.
647
-
648
- #### `schedules update`
649
-
650
- Update an existing schedule:
651
-
652
- ```bash
653
- pyworkflow schedules update sched_abc123 --cron "0 10 * * *"
654
- pyworkflow schedules update sched_abc123 --overlap buffer_one
655
- ```
656
-
657
- **Options:**
658
- | Option | Description |
659
- |--------|-------------|
660
- | `--cron` | New cron expression |
661
- | `--interval` | New interval duration |
662
- | `--overlap` | New overlap policy |
663
-
664
- #### `schedules backfill`
665
-
666
- Backfill missed runs for a schedule:
667
-
668
- ```bash
669
- pyworkflow schedules backfill sched_abc123 \
670
- --start 2024-01-01T00:00:00 \
671
- --end 2024-01-31T23:59:59
672
- ```
673
-
674
- **Options:**
675
- | Option | Required | Description |
676
- |--------|----------|-------------|
677
- | `--start` | Yes | Start time for backfill (ISO format) |
678
- | `--end` | Yes | End time for backfill (ISO format) |
679
-
680
- <Warning>
681
- Backfill creates runs for all scheduled times in the range. For high-frequency schedules, this could create many runs.
682
- </Warning>
683
-
684
- ---
685
-
686
- ### Quickstart Command
687
-
688
- Create a new PyWorkflow project with sample workflows.
689
-
690
- #### `quickstart`
691
-
692
- Scaffold a complete project structure with working examples:
693
-
694
- ```bash
695
- pyworkflow quickstart
696
- ```
697
-
698
- **Options:**
699
- | Option | Description |
700
- |--------|-------------|
701
- | `--non-interactive` | Run without prompts, use defaults |
702
- | `--skip-docker` | Skip Docker services setup |
703
- | `--template TYPE` | Project template: `basic` (default) |
704
- | `--storage TYPE` | Storage backend: `sqlite` or `file` |
705
-
706
- **Examples:**
707
-
708
- ```bash
709
- # Interactive quickstart (recommended for first-time users)
710
- pyworkflow quickstart
711
-
712
- # Non-interactive with defaults
713
- pyworkflow quickstart --non-interactive
714
-
715
- # Without Docker services
716
- pyworkflow quickstart --skip-docker
717
-
718
- # Use file storage instead of SQLite
719
- pyworkflow quickstart --storage file
720
- ```
721
-
722
- **Created Files:**
723
-
724
- ```
725
- myproject/
726
- ├── workflows/
727
- │ ├── __init__.py # Exports all workflows
728
- │ ├── orders.py # process_order workflow
729
- │ └── notifications.py # send_notification workflow
730
- ├── pyworkflow.config.yaml # Configuration
731
- └── docker-compose.yml # Docker services (if enabled)
732
- ```
733
-
734
- <Tip>
735
- Use `pyworkflow quickstart` to bootstrap a new project, then modify the sample
736
- workflows or add your own in the `workflows/` directory.
737
- </Tip>
738
-
739
- ---
740
-
741
- ### Setup Command
742
-
743
- Configure the PyWorkflow environment for an existing project.
744
-
745
- #### `setup`
746
-
747
- Interactive setup that generates configuration and Docker files:
748
-
749
- ```bash
750
- pyworkflow setup
751
- ```
752
-
753
- **Options:**
754
- | Option | Description |
755
- |--------|-------------|
756
- | `--non-interactive` | Run without prompts (use defaults) |
757
- | `--skip-docker` | Skip Docker infrastructure setup |
758
- | `--module PATH` | Workflow module path (e.g., `myapp.workflows`) |
759
- | `--storage TYPE` | Storage backend: `file`, `memory`, or `sqlite` |
760
- | `--storage-path PATH` | Storage path for file/sqlite backends |
761
-
762
- **Examples:**
763
-
764
- ```bash
765
- # Interactive setup (recommended)
766
- pyworkflow setup
767
-
768
- # Non-interactive with defaults
769
- pyworkflow setup --non-interactive
770
-
771
- # Skip Docker setup
772
- pyworkflow setup --skip-docker
773
-
774
- # Specify options directly
775
- pyworkflow setup --module myapp.workflows --storage sqlite
776
- ```
777
-
778
- ## Output Formats
779
-
780
- Control output format with the `--output` flag:
781
-
782
- <Tabs>
783
- <Tab title="Table (Default)">
784
- ```bash
785
- pyworkflow runs list
786
- ```
787
- ```
788
- ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┓
789
- ┃ Run ID ┃ Workflow ┃ Status ┃ Started ┃ Duration ┃
790
- ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━┩
791
- │ run_abc123... │ onboarding │ completed │ 10:30:45 │ 1.2s │
792
- │ run_def456... │ payment │ running │ 10:31:02 │ 0.5s │
793
- └────────────────┴─────────────┴───────────┴─────────────┴──────────┘
794
- ```
795
- </Tab>
796
- <Tab title="JSON">
797
- ```bash
798
- pyworkflow --output json runs list
799
- ```
800
- ```json
801
- [
802
- {
803
- "run_id": "run_abc123...",
804
- "workflow": "onboarding",
805
- "status": "completed",
806
- "started_at": "2025-01-15T10:30:45Z",
807
- "duration": 1.2
808
- }
809
- ]
810
- ```
811
- </Tab>
812
- <Tab title="Plain">
813
- ```bash
814
- pyworkflow --output plain runs list
815
- ```
816
- ```
817
- run_abc123...
818
- run_def456...
819
- ```
820
- </Tab>
821
- </Tabs>
822
-
823
- <Tip>
824
- Use `--output json` for scripting and automation. Use `--output plain` for simple lists suitable for piping to other commands.
825
- </Tip>
826
-
827
- ## Examples
828
-
829
- ### Complete Workflow Lifecycle
830
-
831
- ```bash
832
- # 1. List available workflows
833
- pyworkflow --module myapp.workflows workflows list
834
-
835
- # 2. Get details about a workflow
836
- pyworkflow --module myapp.workflows workflows info onboarding_workflow
837
-
838
- # 3. Run the workflow
839
- pyworkflow --module myapp.workflows workflows run onboarding_workflow \
840
- --arg user_id=user_123
841
-
842
- # Output: Workflow started: run_abc123def456
843
-
844
- # 4. Check the status
845
- pyworkflow runs status run_abc123def456
846
-
847
- # 5. View the event log
848
- pyworkflow runs logs run_abc123def456
849
- ```
850
-
851
- ### Debugging Failed Runs
852
-
853
- ```bash
854
- # Find failed runs
855
- pyworkflow runs list --status failed
856
-
857
- # Check error details (verbose mode for full traceback)
858
- pyworkflow --verbose runs status run_xyz789
859
-
860
- # View events leading to failure
861
- pyworkflow runs logs run_xyz789 --filter failed
862
- ```
863
-
864
- ### Scripting with JSON Output
865
-
866
- ```bash
867
- # Get failed run IDs for batch processing
868
- pyworkflow --output json runs list --status failed | jq -r '.[].run_id'
869
-
870
- # Export workflow list
871
- pyworkflow --output json workflows list > workflows.json
872
- ```
873
-
874
- ### Using Config File
875
-
876
- With a `pyworkflow.toml` in your project:
877
-
878
- ```toml
879
- module = "myapp.workflows"
880
-
881
- [storage]
882
- backend = "file"
883
- path = "./data/workflows"
884
- ```
885
-
886
- Commands become simpler:
887
-
888
- ```bash
889
- # No --module needed
890
- pyworkflow workflows list
891
- pyworkflow workflows run my_workflow --arg foo=bar
892
- ```
893
-
894
- ### Distributed Workflow Execution
895
-
896
- Complete example of running workflows on Celery workers:
897
-
898
- ```bash
899
- # 1. Setup and verify environment
900
- pyworkflow setup --check
901
-
902
- # 2. Start Redis (if not running)
903
- docker run -d -p 6379:6379 redis:7-alpine
904
-
905
- # 3. Start workers (in separate terminals)
906
- pyworkflow worker run --workflow # Workflow orchestration
907
- pyworkflow worker run --step # Step execution
908
- pyworkflow worker run --schedule # Sleep resumption
909
-
910
- # Or start all-in-one worker
911
- pyworkflow worker run
912
-
913
- # 4. Run a workflow (dispatched to Celery)
914
- pyworkflow --module myapp.workflows workflows run my_workflow \
915
- --arg user_id=123
916
-
917
- # 5. Monitor execution
918
- pyworkflow runs list
919
- pyworkflow runs status run_abc123
920
- pyworkflow runs logs run_abc123
921
- ```
922
-
923
- <Tip>
924
- Use `--runtime local` to run workflows in-process without Celery for testing or simple scripts.
925
- </Tip>
926
-
927
- ## Next Steps
928
-
929
- <CardGroup cols={2}>
930
- <Card title="Quick Start" icon="rocket" href="/quickstart">
931
- Get started with PyWorkflow basics.
932
- </Card>
933
- <Card title="Workflows" icon="diagram-project" href="/concepts/workflows">
934
- Learn about workflow concepts and patterns.
935
- </Card>
936
- <Card title="Events" icon="timeline" href="/concepts/events">
937
- Understand event sourcing and replay.
938
- </Card>
939
- <Card title="Deployment" icon="server" href="/guides/deployment">
940
- Deploy workflows to production.
941
- </Card>
942
- </CardGroup>