queueflow 0.2.0__tar.gz → 0.2.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (123) hide show
  1. queueflow-0.2.1/LICENSE +21 -0
  2. queueflow-0.2.1/PKG-INFO +311 -0
  3. queueflow-0.2.1/README.md +270 -0
  4. {queueflow-0.2.0 → queueflow-0.2.1}/pyproject.toml +53 -25
  5. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/__init__.py +1 -1
  6. queueflow-0.2.1/queueflow.egg-info/PKG-INFO +311 -0
  7. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow.egg-info/SOURCES.txt +2 -0
  8. queueflow-0.2.1/queueflow.egg-info/requires.txt +12 -0
  9. {queueflow-0.2.0 → queueflow-0.2.1}/setup.py +2 -2
  10. queueflow-0.2.1/test/test_live.py +167 -0
  11. queueflow-0.2.0/PKG-INFO +0 -26
  12. queueflow-0.2.0/README.md +0 -188
  13. queueflow-0.2.0/queueflow.egg-info/PKG-INFO +0 -26
  14. queueflow-0.2.0/queueflow.egg-info/requires.txt +0 -4
  15. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/__init__.py +0 -0
  16. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/cron_api.py +0 -0
  17. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/dlq_api.py +0 -0
  18. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/health_api.py +0 -0
  19. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/jobs_api.py +0 -0
  20. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/system_api.py +0 -0
  21. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/worker_api.py +0 -0
  22. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api/workflows_api.py +0 -0
  23. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api_client.py +0 -0
  24. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/api_response.py +0 -0
  25. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/configuration.py +0 -0
  26. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/exceptions.py +0 -0
  27. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/facade.py +0 -0
  28. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/__init__.py +0 -0
  29. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/backoff_strategy.py +0 -0
  30. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/complete_job_request.py +0 -0
  31. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_batch_jobs_request.py +0 -0
  32. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_batch_jobs_response.py +0 -0
  33. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_cron_request.py +0 -0
  34. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_cron_response.py +0 -0
  35. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_job_request.py +0 -0
  36. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_job_response.py +0 -0
  37. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_workflow_request.py +0 -0
  38. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/create_workflow_response.py +0 -0
  39. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/cron_schedule.py +0 -0
  40. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/dead_letter.py +0 -0
  41. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/error_body.py +0 -0
  42. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/fail_job_request.py +0 -0
  43. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/health_status.py +0 -0
  44. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/heartbeat_request.py +0 -0
  45. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/heartbeat_response.py +0 -0
  46. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/job.py +0 -0
  47. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/job_config.py +0 -0
  48. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/job_config_request.py +0 -0
  49. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/job_status.py +0 -0
  50. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/lease_jobs_request.py +0 -0
  51. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/lease_jobs_response.py +0 -0
  52. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/leased_job.py +0 -0
  53. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/list_crons_response.py +0 -0
  54. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/list_dead_letters_response.py +0 -0
  55. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/list_jobs_response.py +0 -0
  56. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/list_workflows_response.py +0 -0
  57. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/on_failure.py +0 -0
  58. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/on_success.py +0 -0
  59. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/ready_status.py +0 -0
  60. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/replay_dead_letter_response.py +0 -0
  61. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/stats_snapshot.py +0 -0
  62. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/step_status.py +0 -0
  63. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/tasks_response.py +0 -0
  64. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow.py +0 -0
  65. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow_diagram_response.py +0 -0
  66. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow_status.py +0 -0
  67. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow_step.py +0 -0
  68. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow_step_state.py +0 -0
  69. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/models/workflow_step_states_response.py +0 -0
  70. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/py.typed +0 -0
  71. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow/rest.py +0 -0
  72. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow.egg-info/dependency_links.txt +0 -0
  73. {queueflow-0.2.0 → queueflow-0.2.1}/queueflow.egg-info/top_level.txt +0 -0
  74. {queueflow-0.2.0 → queueflow-0.2.1}/setup.cfg +0 -0
  75. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_backoff_strategy.py +0 -0
  76. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_complete_job_request.py +0 -0
  77. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_batch_jobs_request.py +0 -0
  78. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_batch_jobs_response.py +0 -0
  79. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_cron_request.py +0 -0
  80. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_cron_response.py +0 -0
  81. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_job_request.py +0 -0
  82. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_job_response.py +0 -0
  83. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_workflow_request.py +0 -0
  84. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_create_workflow_response.py +0 -0
  85. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_cron_api.py +0 -0
  86. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_cron_schedule.py +0 -0
  87. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_dead_letter.py +0 -0
  88. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_dlq_api.py +0 -0
  89. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_error_body.py +0 -0
  90. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_facade.py +0 -0
  91. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_fail_job_request.py +0 -0
  92. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_health_api.py +0 -0
  93. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_health_status.py +0 -0
  94. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_heartbeat_request.py +0 -0
  95. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_heartbeat_response.py +0 -0
  96. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_job.py +0 -0
  97. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_job_config.py +0 -0
  98. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_job_config_request.py +0 -0
  99. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_job_status.py +0 -0
  100. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_jobs_api.py +0 -0
  101. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_lease_jobs_request.py +0 -0
  102. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_lease_jobs_response.py +0 -0
  103. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_leased_job.py +0 -0
  104. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_list_crons_response.py +0 -0
  105. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_list_dead_letters_response.py +0 -0
  106. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_list_jobs_response.py +0 -0
  107. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_list_workflows_response.py +0 -0
  108. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_on_failure.py +0 -0
  109. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_on_success.py +0 -0
  110. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_ready_status.py +0 -0
  111. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_replay_dead_letter_response.py +0 -0
  112. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_stats_snapshot.py +0 -0
  113. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_step_status.py +0 -0
  114. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_system_api.py +0 -0
  115. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_tasks_response.py +0 -0
  116. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_worker_api.py +0 -0
  117. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow.py +0 -0
  118. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow_diagram_response.py +0 -0
  119. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow_status.py +0 -0
  120. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow_step.py +0 -0
  121. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow_step_state.py +0 -0
  122. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflow_step_states_response.py +0 -0
  123. {queueflow-0.2.0 → queueflow-0.2.1}/test/test_workflows_api.py +0 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2025 QueueFlow
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,311 @@
1
+ Metadata-Version: 2.4
2
+ Name: queueflow
3
+ Version: 0.2.1
4
+ Summary: Python client for QueueFlow, a PostgreSQL-native distributed job queue and workflow engine.
5
+ Home-page: https://github.com/elision-labs/queueflow-sdk-python
6
+ Author: QueueFlow
7
+ Author-email: QueueFlow <team@queueflow.dev>
8
+ License-Expression: MIT
9
+ Project-URL: Homepage, https://queueflow.dev
10
+ Project-URL: Documentation, https://docs.queueflow.dev
11
+ Project-URL: Repository, https://github.com/elision-labs/queueflow-sdk-python
12
+ Project-URL: Server, https://github.com/elision-labs/queueflow-core
13
+ Keywords: queue,jobs,workflow,dag,postgres,queueflow,background-jobs
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.9
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Topic :: System :: Distributed Computing
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.9
25
+ Description-Content-Type: text/markdown
26
+ License-File: LICENSE
27
+ Requires-Dist: urllib3<3.0.0,>=1.25.3
28
+ Requires-Dist: python-dateutil>=2.8.2
29
+ Requires-Dist: pydantic>=2
30
+ Requires-Dist: typing-extensions>=4.7.1
31
+ Provides-Extra: dev
32
+ Requires-Dist: pytest>=7.2.1; extra == "dev"
33
+ Requires-Dist: pytest-cov>=2.8.1; extra == "dev"
34
+ Requires-Dist: tox>=3.9.0; extra == "dev"
35
+ Requires-Dist: flake8>=4.0.0; extra == "dev"
36
+ Requires-Dist: types-python-dateutil>=2.8.19.14; extra == "dev"
37
+ Requires-Dist: mypy>=1.5; extra == "dev"
38
+ Dynamic: author
39
+ Dynamic: home-page
40
+ Dynamic: license-file
41
+
42
+ # queueflow
43
+
44
+ Python client for [QueueFlow](https://queueflow.dev), a PostgreSQL-native distributed job queue and
45
+ workflow engine.
46
+
47
+ - **Typed end-to-end**: request/response models (pydantic v2) mirror the server's OpenAPI 3.1 spec.
48
+ - **Ergonomic**: `qf.create_job(...)`, `wait_for_*` pollers, and a workflow builder DSL.
49
+ - **Worker runtime**: `qf.run_worker(...)` runs Python task handlers against a remote server.
50
+ - **Thin facade over a generated core**: the facade (`queueflow.facade`) adds what codegen cannot;
51
+ the generated `queueflow.api.*` clients and `queueflow.models.*` stay available for everything else.
52
+
53
+ Python 3.9 or newer.
54
+
55
+ ## Install
56
+
57
+ ```bash
58
+ pip install queueflow
59
+ ```
60
+
61
+ ## Quick start
62
+
63
+ ```python
64
+ from queueflow.facade import QueueFlow, wf
65
+
66
+ qf = QueueFlow("http://localhost:8000", "dev")
67
+
68
+ # Enqueue a job and wait for the result.
69
+ job = qf.create_job("echo", payload={"hello": "world"}, max_retries=3, timeout=30)
70
+ done = qf.wait_for_job(job.id)
71
+ print(done.status, done.result) # completed {'hello': 'world', 'echoed': True}
72
+
73
+ # Declare and run a DAG workflow.
74
+ dag = (
75
+ wf("etl")
76
+ .step("extract", "echo")
77
+ .step("transform", "echo", after=["extract"])
78
+ .step("load", "echo", after=["transform"], on_failure="halt")
79
+ )
80
+ workflow = qf.create_workflow(dag)
81
+ finished = qf.wait_for_workflow(workflow.id)
82
+ print(finished.status, finished.context)
83
+ ```
84
+
85
+ `echo` is a handler built into the server; it returns the payload plus `"echoed": true`.
86
+
87
+ ## API
88
+
89
+ ### Client
90
+
91
+ ```python
92
+ qf = QueueFlow(
93
+ base_url, # required, e.g. "http://localhost:8000"
94
+ token, # required: tenant bearer token (any non-empty token on the dev server)
95
+ worker_token=None, # credential for qf.worker and run_worker (defaults to token; dev mode only)
96
+ )
97
+ ```
98
+
99
+ The generated per-tag clients hang off the facade for anything the helpers do not cover:
100
+ `qf.jobs`, `qf.workflows`, `qf.worker`, `qf.cron`, `qf.dlq`, `qf.system`, `qf.health_api`.
101
+
102
+ ### Jobs
103
+
104
+ | Method | Description |
105
+ | --- | --- |
106
+ | `qf.create_job(task, payload=None, priority=None, max_retries=None, timeout=None, queue=None, retry_backoff=None, retry_delay_secs=None, retry_max_delay_secs=None, jitter_factor=None, idempotency_key=None, run_at=None)` | Enqueue a job and return the created `Job`. |
107
+ | `qf.wait_for_job(job_id, timeout=60.0, interval=0.5)` | Poll until `completed` / `failed` / `cancelled`; raises `WaitTimeout`. |
108
+ | `qf.jobs.get_job(id)` | Fetch a job. |
109
+ | `qf.jobs.list_jobs(status=..., queue=..., limit=..., offset=..., order_by=..., include_total=..., cursor=..., created_after=..., created_before=...)` | List jobs. |
110
+ | `qf.jobs.cancel_job(id)` | Cancel a job. |
111
+ | `qf.jobs.create_batch_jobs(CreateBatchJobsRequest(...))` | Enqueue up to 1000 jobs at once. |
112
+
113
+ `idempotency_key` is sent as the `Idempotency-Key` header: re-submitting the same key returns the
114
+ original job instead of creating a duplicate. `run_at` (a `datetime`) delays the first run.
115
+
116
+ List responses carry `next_cursor` when there are more pages; pass it back as `cursor` for keyset
117
+ pagination (cheaper than deep `offset`).
118
+
119
+ ### Workflows
120
+
121
+ | Method | Description |
122
+ | --- | --- |
123
+ | `qf.create_workflow(builder_or_request)` | Create a workflow from a `wf()` builder or a raw `CreateWorkflowRequest`. |
124
+ | `qf.wait_for_workflow(workflow_id, timeout=60.0, interval=0.5)` | Poll until a terminal workflow state. |
125
+ | `qf.workflows.get_workflow(id)` / `list_workflows(...)` / `cancel_workflow(id)` | Fetch / list / cancel. |
126
+ | `qf.workflows.get_workflow_step_states(id)` | Runtime status of every step, in declaration order. |
127
+ | `qf.workflows.get_workflow_diagram(id)` | Mermaid (`graph TD`) diagram of the DAG. |
128
+
129
+ ### Workflow builder
130
+
131
+ ```python
132
+ from queueflow.facade import wf
133
+
134
+ dag = (
135
+ wf("order_123")
136
+ .step("validate", "validate_order")
137
+ .step("pay", "process_payment", after=["validate"])
138
+ .step("ship", "create_shipment", after=["pay"], on_failure="continue")
139
+ .context(source="web")
140
+ )
141
+ # .build() runs locally first: duplicate names, dangling deps, and cycles raise
142
+ # WorkflowValidationError before any network round-trip.
143
+ ```
144
+
145
+ `step()` also accepts `payload=`, `on_success=`, and `config=` (a `models.JobConfig` override).
146
+
147
+ ### Worker
148
+
149
+ Run task handlers in this process against a remote QueueFlow server:
150
+
151
+ ```python
152
+ import threading
153
+ from queueflow.facade import QueueFlow, NonRetryableError
154
+
155
+ qf = QueueFlow("http://localhost:8000", "tenant-key", worker_token="worker-secret")
156
+
157
+ def send_email(job):
158
+ if not job.payload.get("to"):
159
+ raise NonRetryableError("no recipient") # straight to the dead-letter queue
160
+ ... # raising anything else retries per policy
161
+ return {"sent": True}
162
+
163
+ def transcode(job, ctx):
164
+ for chunk in chunks(job.payload):
165
+ if ctx.cancelled: # lease lost / cancelled mid-run: stop,
166
+ return {} # the server owns the outcome now
167
+ process(chunk)
168
+ return {"ok": True}
169
+
170
+ stop = threading.Event() # set it (e.g. from a signal handler) to drain
171
+ qf.run_worker("orders", {"send_email": send_email, "transcode": transcode}, stop=stop)
172
+ ```
173
+
174
+ `run_worker(queue, handlers, lease_secs=30, wait_secs=20, stop=None, on_error=None)` leases one
175
+ job at a time, heartbeats each in-flight job at half the lease interval, stops reporting when the
176
+ lease is lost, and applies the server's retry policy on errors. Handlers take `(job)` or
177
+ `(job, ctx)` and return the result dict (or `None` for `{}`). Delivery is at-least-once, so make
178
+ handlers idempotent. The low-level calls (`qf.worker.lease_jobs`, `heartbeat_job`, `complete_job`,
179
+ `fail_job`) are also exposed.
180
+
181
+ Rules the runtime enforces, and that any hand-rolled loop over `qf.worker` must honour too:
182
+
183
+ 1. Worker routes authenticate with the **worker token**, not a tenant token. Pass it as
184
+ `worker_token=`; without it `qf.worker` reuses the tenant token, which only works in the
185
+ server's development mode. `run_worker` raises on 401/403 rather than spinning.
186
+ 2. Every in-flight job is heartbeated at half its lease interval. A heartbeat showing the job no
187
+ longer running (or an HTTP 409) abandons reporting and flips `ctx.cancelled`.
188
+ 3. Raise `NonRetryableError` (or any exception with `retryable = False`) for permanent failures so
189
+ they dead-letter immediately. A failed outcome *report* is never converted into a job failure;
190
+ the lease expires and the engine redelivers.
191
+
192
+ ### Cron
193
+
194
+ | Method | Description |
195
+ | --- | --- |
196
+ | `qf.create_cron(name, schedule, task, payload=None, queue=None)` | Register a recurring enqueue (5-field crontab, UTC); returns the schedule id. |
197
+ | `qf.cron.get_cron(id)` / `list_crons(...)` / `delete_cron(id)` | Fetch / list / delete. |
198
+ | `qf.cron.pause_cron(id)` / `resume_cron(id)` | Stop firings / resume at the next future occurrence. |
199
+
200
+ ### Dead letters
201
+
202
+ | Method | Description |
203
+ | --- | --- |
204
+ | `qf.dlq.list_dead_letters(...)` / `get_dead_letter(id)` | Inspect terminally-failed jobs. |
205
+ | `qf.replay_dead_letter(id)` | Re-run one as a fresh job and return it (at most once; a second replay is a 409). |
206
+
207
+ ### System
208
+
209
+ ```python
210
+ qf.system.get_stats() # engine counters
211
+ qf.system.list_tasks() # registered task handler names
212
+ qf.health_api.get_health()
213
+ qf.health_api.get_ready()
214
+ ```
215
+
216
+ ## Authentication
217
+
218
+ Every request carries `Authorization: Bearer <token>`. Two credentials exist:
219
+
220
+ - The **tenant token** (`token`) authenticates the job, workflow, cron, DLQ and system routes. On a
221
+ server started without `--api-keys`, any non-empty token is accepted (development mode).
222
+ - The **worker token** (`worker_token`) authenticates the worker-protocol routes (lease, heartbeat,
223
+ complete, fail). Configure it on the server with `--worker-token` / `QUEUEFLOW_WORKER_TOKEN`.
224
+ It is a separate secret, never a tenant token.
225
+
226
+ ## Error handling
227
+
228
+ Generated calls raise `queueflow.exceptions.ApiException` (with `.status`, `.reason`, `.body`) and
229
+ its subclasses `BadRequestException` (400), `UnauthorizedException` (401), `ForbiddenException`
230
+ (403), `NotFoundException` (404), `ServiceException` (5xx). The facade adds `WaitTimeout` (a
231
+ `wait_for_*` deadline passed), `WorkflowValidationError` (a bad DAG, raised before any request),
232
+ and `NonRetryableError` (raise from a handler to dead-letter immediately).
233
+
234
+ ```python
235
+ from queueflow.exceptions import ApiException, NotFoundException
236
+ from queueflow.facade import WaitTimeout
237
+
238
+ try:
239
+ job = qf.wait_for_job("missing", timeout=10)
240
+ except NotFoundException:
241
+ ...
242
+ except WaitTimeout:
243
+ ...
244
+ except ApiException as err:
245
+ print(err.status, err.body)
246
+ ```
247
+
248
+ ## Known limitation: `stream_job_events`
249
+
250
+ `JobsApi.stream_job_events` cannot stream: it buffers the whole SSE response until the server
251
+ closes it (terminal status or the 15-minute cap). Use `wait_for_job` (polling) instead, or the
252
+ `stream_job_events_without_preload_content` variant with a hand-rolled SSE parser.
253
+
254
+ ## Conformance tests
255
+
256
+ `test/test_live.py` runs the facade against a real QueueFlow server (started with the default
257
+ `queueflow serve`, i.e. `--mode all`, so the built-in `echo` handler is registered). It creates
258
+ and waits on an `echo` job, checks idempotent re-creation, runs a two-step workflow, exercises the
259
+ cron and dead-letter endpoints, reads stats, and runs `run_worker` against a dedicated queue. The
260
+ tests are marked `live` and skip themselves unless `QUEUEFLOW_URL` is set, so plain `pytest` and
261
+ CI stay offline.
262
+
263
+ ```bash
264
+ QUEUEFLOW_URL=http://localhost:8000 \
265
+ QUEUEFLOW_TOKEN=dev \
266
+ QUEUEFLOW_WORKER_TOKEN=worker-secret \
267
+ pytest -m live
268
+ ```
269
+
270
+ `QUEUEFLOW_TOKEN` is the tenant token (default `dev`). `QUEUEFLOW_WORKER_TOKEN` is the server's
271
+ `--worker-token`; it defaults to the tenant token, which only works when the server runs without
272
+ one.
273
+
274
+ ## Architecture
275
+
276
+ This package is a thin hand-written **facade** over a **generated core**:
277
+
278
+ ```
279
+ queueflow-sdk-python/
280
+ ├── queueflow/
281
+ │ ├── facade.py the hand-written facade (QueueFlow, wf, run_worker, errors)
282
+ │ ├── api/ generated per-tag clients (JobsApi, WorkflowsApi, WorkerApi, ...)
283
+ │ ├── models/ generated pydantic models
284
+ │ └── api_client.py, configuration.py, rest.py, exceptions.py generated transport
285
+ ├── docs/ generated per-endpoint and per-model reference
286
+ └── test/ test_facade.py (offline), test_live.py (conformance), generated model tests
287
+ ```
288
+
289
+ The core is generated by openapi-generator from the server's
290
+ [OpenAPI spec](https://github.com/elision-labs/queueflow-core/blob/main/spec/openapi.yaml)
291
+ (`scripts/generate-sdks.sh python` in queueflow-core). The facade is injected at generation time
292
+ from `sdk-templates/python/facade.mustache` in that repo, so edit the template, not
293
+ `queueflow/facade.py`. Per-endpoint reference for the generated layer lives in [`docs/`](./docs).
294
+
295
+ ### Development
296
+
297
+ ```bash
298
+ pip install -e . -r test-requirements.txt
299
+ pytest # offline tests (facade + generated model tests)
300
+ mypy # type-check queueflow/ and test/
301
+ ```
302
+
303
+ ## Links
304
+
305
+ - Documentation: [docs.queueflow.dev](https://docs.queueflow.dev)
306
+ - Server: [github.com/elision-labs/queueflow-core](https://github.com/elision-labs/queueflow-core)
307
+ - This SDK: [github.com/elision-labs/queueflow-sdk-python](https://github.com/elision-labs/queueflow-sdk-python)
308
+
309
+ ## License
310
+
311
+ [MIT](./LICENSE)
@@ -0,0 +1,270 @@
1
+ # queueflow
2
+
3
+ Python client for [QueueFlow](https://queueflow.dev), a PostgreSQL-native distributed job queue and
4
+ workflow engine.
5
+
6
+ - **Typed end-to-end**: request/response models (pydantic v2) mirror the server's OpenAPI 3.1 spec.
7
+ - **Ergonomic**: `qf.create_job(...)`, `wait_for_*` pollers, and a workflow builder DSL.
8
+ - **Worker runtime**: `qf.run_worker(...)` runs Python task handlers against a remote server.
9
+ - **Thin facade over a generated core**: the facade (`queueflow.facade`) adds what codegen cannot;
10
+ the generated `queueflow.api.*` clients and `queueflow.models.*` stay available for everything else.
11
+
12
+ Python 3.9 or newer.
13
+
14
+ ## Install
15
+
16
+ ```bash
17
+ pip install queueflow
18
+ ```
19
+
20
+ ## Quick start
21
+
22
+ ```python
23
+ from queueflow.facade import QueueFlow, wf
24
+
25
+ qf = QueueFlow("http://localhost:8000", "dev")
26
+
27
+ # Enqueue a job and wait for the result.
28
+ job = qf.create_job("echo", payload={"hello": "world"}, max_retries=3, timeout=30)
29
+ done = qf.wait_for_job(job.id)
30
+ print(done.status, done.result) # completed {'hello': 'world', 'echoed': True}
31
+
32
+ # Declare and run a DAG workflow.
33
+ dag = (
34
+ wf("etl")
35
+ .step("extract", "echo")
36
+ .step("transform", "echo", after=["extract"])
37
+ .step("load", "echo", after=["transform"], on_failure="halt")
38
+ )
39
+ workflow = qf.create_workflow(dag)
40
+ finished = qf.wait_for_workflow(workflow.id)
41
+ print(finished.status, finished.context)
42
+ ```
43
+
44
+ `echo` is a handler built into the server; it returns the payload plus `"echoed": true`.
45
+
46
+ ## API
47
+
48
+ ### Client
49
+
50
+ ```python
51
+ qf = QueueFlow(
52
+ base_url, # required, e.g. "http://localhost:8000"
53
+ token, # required: tenant bearer token (any non-empty token on the dev server)
54
+ worker_token=None, # credential for qf.worker and run_worker (defaults to token; dev mode only)
55
+ )
56
+ ```
57
+
58
+ The generated per-tag clients hang off the facade for anything the helpers do not cover:
59
+ `qf.jobs`, `qf.workflows`, `qf.worker`, `qf.cron`, `qf.dlq`, `qf.system`, `qf.health_api`.
60
+
61
+ ### Jobs
62
+
63
+ | Method | Description |
64
+ | --- | --- |
65
+ | `qf.create_job(task, payload=None, priority=None, max_retries=None, timeout=None, queue=None, retry_backoff=None, retry_delay_secs=None, retry_max_delay_secs=None, jitter_factor=None, idempotency_key=None, run_at=None)` | Enqueue a job and return the created `Job`. |
66
+ | `qf.wait_for_job(job_id, timeout=60.0, interval=0.5)` | Poll until `completed` / `failed` / `cancelled`; raises `WaitTimeout`. |
67
+ | `qf.jobs.get_job(id)` | Fetch a job. |
68
+ | `qf.jobs.list_jobs(status=..., queue=..., limit=..., offset=..., order_by=..., include_total=..., cursor=..., created_after=..., created_before=...)` | List jobs. |
69
+ | `qf.jobs.cancel_job(id)` | Cancel a job. |
70
+ | `qf.jobs.create_batch_jobs(CreateBatchJobsRequest(...))` | Enqueue up to 1000 jobs at once. |
71
+
72
+ `idempotency_key` is sent as the `Idempotency-Key` header: re-submitting the same key returns the
73
+ original job instead of creating a duplicate. `run_at` (a `datetime`) delays the first run.
74
+
75
+ List responses carry `next_cursor` when there are more pages; pass it back as `cursor` for keyset
76
+ pagination (cheaper than deep `offset`).
77
+
78
+ ### Workflows
79
+
80
+ | Method | Description |
81
+ | --- | --- |
82
+ | `qf.create_workflow(builder_or_request)` | Create a workflow from a `wf()` builder or a raw `CreateWorkflowRequest`. |
83
+ | `qf.wait_for_workflow(workflow_id, timeout=60.0, interval=0.5)` | Poll until a terminal workflow state. |
84
+ | `qf.workflows.get_workflow(id)` / `list_workflows(...)` / `cancel_workflow(id)` | Fetch / list / cancel. |
85
+ | `qf.workflows.get_workflow_step_states(id)` | Runtime status of every step, in declaration order. |
86
+ | `qf.workflows.get_workflow_diagram(id)` | Mermaid (`graph TD`) diagram of the DAG. |
87
+
88
+ ### Workflow builder
89
+
90
+ ```python
91
+ from queueflow.facade import wf
92
+
93
+ dag = (
94
+ wf("order_123")
95
+ .step("validate", "validate_order")
96
+ .step("pay", "process_payment", after=["validate"])
97
+ .step("ship", "create_shipment", after=["pay"], on_failure="continue")
98
+ .context(source="web")
99
+ )
100
+ # .build() runs locally first: duplicate names, dangling deps, and cycles raise
101
+ # WorkflowValidationError before any network round-trip.
102
+ ```
103
+
104
+ `step()` also accepts `payload=`, `on_success=`, and `config=` (a `models.JobConfig` override).
105
+
106
+ ### Worker
107
+
108
+ Run task handlers in this process against a remote QueueFlow server:
109
+
110
+ ```python
111
+ import threading
112
+ from queueflow.facade import QueueFlow, NonRetryableError
113
+
114
+ qf = QueueFlow("http://localhost:8000", "tenant-key", worker_token="worker-secret")
115
+
116
+ def send_email(job):
117
+ if not job.payload.get("to"):
118
+ raise NonRetryableError("no recipient") # straight to the dead-letter queue
119
+ ... # raising anything else retries per policy
120
+ return {"sent": True}
121
+
122
+ def transcode(job, ctx):
123
+ for chunk in chunks(job.payload):
124
+ if ctx.cancelled: # lease lost / cancelled mid-run: stop,
125
+ return {} # the server owns the outcome now
126
+ process(chunk)
127
+ return {"ok": True}
128
+
129
+ stop = threading.Event() # set it (e.g. from a signal handler) to drain
130
+ qf.run_worker("orders", {"send_email": send_email, "transcode": transcode}, stop=stop)
131
+ ```
132
+
133
+ `run_worker(queue, handlers, lease_secs=30, wait_secs=20, stop=None, on_error=None)` leases one
134
+ job at a time, heartbeats each in-flight job at half the lease interval, stops reporting when the
135
+ lease is lost, and applies the server's retry policy on errors. Handlers take `(job)` or
136
+ `(job, ctx)` and return the result dict (or `None` for `{}`). Delivery is at-least-once, so make
137
+ handlers idempotent. The low-level calls (`qf.worker.lease_jobs`, `heartbeat_job`, `complete_job`,
138
+ `fail_job`) are also exposed.
139
+
140
+ Rules the runtime enforces, and that any hand-rolled loop over `qf.worker` must honour too:
141
+
142
+ 1. Worker routes authenticate with the **worker token**, not a tenant token. Pass it as
143
+ `worker_token=`; without it `qf.worker` reuses the tenant token, which only works in the
144
+ server's development mode. `run_worker` raises on 401/403 rather than spinning.
145
+ 2. Every in-flight job is heartbeated at half its lease interval. A heartbeat showing the job no
146
+ longer running (or an HTTP 409) abandons reporting and flips `ctx.cancelled`.
147
+ 3. Raise `NonRetryableError` (or any exception with `retryable = False`) for permanent failures so
148
+ they dead-letter immediately. A failed outcome *report* is never converted into a job failure;
149
+ the lease expires and the engine redelivers.
150
+
151
+ ### Cron
152
+
153
+ | Method | Description |
154
+ | --- | --- |
155
+ | `qf.create_cron(name, schedule, task, payload=None, queue=None)` | Register a recurring enqueue (5-field crontab, UTC); returns the schedule id. |
156
+ | `qf.cron.get_cron(id)` / `list_crons(...)` / `delete_cron(id)` | Fetch / list / delete. |
157
+ | `qf.cron.pause_cron(id)` / `resume_cron(id)` | Stop firings / resume at the next future occurrence. |
158
+
159
+ ### Dead letters
160
+
161
+ | Method | Description |
162
+ | --- | --- |
163
+ | `qf.dlq.list_dead_letters(...)` / `get_dead_letter(id)` | Inspect terminally-failed jobs. |
164
+ | `qf.replay_dead_letter(id)` | Re-run one as a fresh job and return it (at most once; a second replay is a 409). |
165
+
166
+ ### System
167
+
168
+ ```python
169
+ qf.system.get_stats() # engine counters
170
+ qf.system.list_tasks() # registered task handler names
171
+ qf.health_api.get_health()
172
+ qf.health_api.get_ready()
173
+ ```
174
+
175
+ ## Authentication
176
+
177
+ Every request carries `Authorization: Bearer <token>`. Two credentials exist:
178
+
179
+ - The **tenant token** (`token`) authenticates the job, workflow, cron, DLQ and system routes. On a
180
+ server started without `--api-keys`, any non-empty token is accepted (development mode).
181
+ - The **worker token** (`worker_token`) authenticates the worker-protocol routes (lease, heartbeat,
182
+ complete, fail). Configure it on the server with `--worker-token` / `QUEUEFLOW_WORKER_TOKEN`.
183
+ It is a separate secret, never a tenant token.
184
+
185
+ ## Error handling
186
+
187
+ Generated calls raise `queueflow.exceptions.ApiException` (with `.status`, `.reason`, `.body`) and
188
+ its subclasses `BadRequestException` (400), `UnauthorizedException` (401), `ForbiddenException`
189
+ (403), `NotFoundException` (404), `ServiceException` (5xx). The facade adds `WaitTimeout` (a
190
+ `wait_for_*` deadline passed), `WorkflowValidationError` (a bad DAG, raised before any request),
191
+ and `NonRetryableError` (raise from a handler to dead-letter immediately).
192
+
193
+ ```python
194
+ from queueflow.exceptions import ApiException, NotFoundException
195
+ from queueflow.facade import WaitTimeout
196
+
197
+ try:
198
+ job = qf.wait_for_job("missing", timeout=10)
199
+ except NotFoundException:
200
+ ...
201
+ except WaitTimeout:
202
+ ...
203
+ except ApiException as err:
204
+ print(err.status, err.body)
205
+ ```
206
+
207
+ ## Known limitation: `stream_job_events`
208
+
209
+ `JobsApi.stream_job_events` cannot stream: it buffers the whole SSE response until the server
210
+ closes it (terminal status or the 15-minute cap). Use `wait_for_job` (polling) instead, or the
211
+ `stream_job_events_without_preload_content` variant with a hand-rolled SSE parser.
212
+
213
+ ## Conformance tests
214
+
215
+ `test/test_live.py` runs the facade against a real QueueFlow server (started with the default
216
+ `queueflow serve`, i.e. `--mode all`, so the built-in `echo` handler is registered). It creates
217
+ and waits on an `echo` job, checks idempotent re-creation, runs a two-step workflow, exercises the
218
+ cron and dead-letter endpoints, reads stats, and runs `run_worker` against a dedicated queue. The
219
+ tests are marked `live` and skip themselves unless `QUEUEFLOW_URL` is set, so plain `pytest` and
220
+ CI stay offline.
221
+
222
+ ```bash
223
+ QUEUEFLOW_URL=http://localhost:8000 \
224
+ QUEUEFLOW_TOKEN=dev \
225
+ QUEUEFLOW_WORKER_TOKEN=worker-secret \
226
+ pytest -m live
227
+ ```
228
+
229
+ `QUEUEFLOW_TOKEN` is the tenant token (default `dev`). `QUEUEFLOW_WORKER_TOKEN` is the server's
230
+ `--worker-token`; it defaults to the tenant token, which only works when the server runs without
231
+ one.
232
+
233
+ ## Architecture
234
+
235
+ This package is a thin hand-written **facade** over a **generated core**:
236
+
237
+ ```
238
+ queueflow-sdk-python/
239
+ ├── queueflow/
240
+ │ ├── facade.py the hand-written facade (QueueFlow, wf, run_worker, errors)
241
+ │ ├── api/ generated per-tag clients (JobsApi, WorkflowsApi, WorkerApi, ...)
242
+ │ ├── models/ generated pydantic models
243
+ │ └── api_client.py, configuration.py, rest.py, exceptions.py generated transport
244
+ ├── docs/ generated per-endpoint and per-model reference
245
+ └── test/ test_facade.py (offline), test_live.py (conformance), generated model tests
246
+ ```
247
+
248
+ The core is generated by openapi-generator from the server's
249
+ [OpenAPI spec](https://github.com/elision-labs/queueflow-core/blob/main/spec/openapi.yaml)
250
+ (`scripts/generate-sdks.sh python` in queueflow-core). The facade is injected at generation time
251
+ from `sdk-templates/python/facade.mustache` in that repo, so edit the template, not
252
+ `queueflow/facade.py`. Per-endpoint reference for the generated layer lives in [`docs/`](./docs).
253
+
254
+ ### Development
255
+
256
+ ```bash
257
+ pip install -e . -r test-requirements.txt
258
+ pytest # offline tests (facade + generated model tests)
259
+ mypy # type-check queueflow/ and test/
260
+ ```
261
+
262
+ ## Links
263
+
264
+ - Documentation: [docs.queueflow.dev](https://docs.queueflow.dev)
265
+ - Server: [github.com/elision-labs/queueflow-core](https://github.com/elision-labs/queueflow-core)
266
+ - This SDK: [github.com/elision-labs/queueflow-sdk-python](https://github.com/elision-labs/queueflow-sdk-python)
267
+
268
+ ## License
269
+
270
+ [MIT](./LICENSE)