dbworker 0.0.5__tar.gz → 0.0.7__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,109 @@
1
+ Metadata-Version: 2.4
2
+ Name: dbworker
3
+ Version: 0.0.7
4
+ Summary: Portable, single-script Python/SQLAlchemy background worker for SQLite, PostgreSQL and MySQL — no Redis, broker or enqueue
5
+ License: MIT
6
+ License-File: LICENSE
7
+ Author: Ziyang Song
8
+ Requires-Python: >=3.12,<4.0
9
+ Classifier: License :: OSI Approved :: MIT License
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.12
12
+ Classifier: Programming Language :: Python :: 3.13
13
+ Classifier: Programming Language :: Python :: 3.14
14
+ Requires-Dist: sqlalchemy (>=2.0,<3.0)
15
+ Project-URL: Repository, https://github.com/zysilm/dbworker
16
+ Description-Content-Type: text/markdown
17
+
18
+ # dbworker
19
+
20
+ **Your database rows are already the queue.**
21
+
22
+ A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
23
+
24
+ ```text
25
+ Celery / Huey / Graphile Worker:
26
+ event → job → worker
27
+
28
+ dbworker:
29
+ row → worker
30
+ ```
31
+
32
+ Requires Python 3.12+. Install into your application:
33
+
34
+ ```sh
35
+ poetry add dbworker
36
+ ```
37
+
38
+ ## Usage
39
+
40
+ Register a handler for an existing SQLAlchemy model. DBWorker discovers eligible
41
+ rows, runs the handler in a child process, and commits its database changes with
42
+ the execution status.
43
+
44
+ ```python
45
+ from sqlalchemy import select
46
+ from sqlalchemy.orm import Session
47
+ from dbworker import Coordinator, Finished
48
+
49
+ coordinator = Coordinator(session_factory, database_url=database_url)
50
+
51
+ @coordinator.transactional_worker(
52
+ name="process",
53
+ source=YourModel,
54
+ eligible=lambda: (
55
+ select(YourModel)
56
+ .where(YourModel.your_bool_property.is_(True))
57
+ .order_by(YourModel.id)
58
+ ),
59
+ concurrency=4,
60
+ )
61
+ def process(source: YourModel, session: Session) -> Finished:
62
+ # Apply your application logic using the supplied session.
63
+ return Finished()
64
+
65
+ if __name__ == "__main__":
66
+ coordinator.create_worker_tables()
67
+ coordinator.start()
68
+ ```
69
+
70
+ `YourModel`, `session_factory`, and `database_url` come from your application.
71
+ `eligible` selects which rows can be claimed and in what order.
72
+ It is optional: omitting it selects from the whole model. DBWorker automatically
73
+ excludes completed, failed, and actively claimed work.
74
+ Return `Finished()` when processing is complete; no enqueue call is needed.
75
+
76
+ See the [Usage Instructions](https://github.com/zysilm/dbworker/blob/main/doc/usage.md) for eligibility queries, incremental
77
+ processing, service startup and shutdown, status tracking, dependencies,
78
+ failure handling, and configuration.
79
+
80
+ ## Examples
81
+
82
+ - [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
83
+ - [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
84
+ - [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
85
+
86
+ ## License
87
+
88
+ [MIT](LICENSE) © 2026 Ziyang Song.
89
+
90
+ <!-- benchmark-results:start -->
91
+ ## Benchmark Results
92
+
93
+ Median wall time in seconds; **bold** marks the faster backend. Each experiment runs in a fresh Docker container on its own GitHub-hosted Ubuntu VM.
94
+
95
+ | Experiment | Scenario | Celery (s) | DBWorker (s) | Celery / DBWorker |
96
+ |---|---|---:|---:|---:|
97
+ | imagededup | build | **4.885** | 5.939 | 0.82 |
98
+ | imagededup | comparison | 78.031 | **74.414** | 1.05 |
99
+ | imagededup | mixed | **85.559** | 86.142 | 0.99 |
100
+ | superset | sql_lab_group_by | **2.045** | 2.070 | 0.99 |
101
+ | saleor | product_csv_export | **1.912** | 2.272 | 0.84 |
102
+ | paperless_ngx | scanned_image_ingestion | 40.368 | **40.143** | 1.01 |
103
+ | posthog | pre_rendered_notification_smtp | 2.006 | **1.511** | 1.33 |
104
+ | sentry | historical_smtp_email | 157.003 | **150.927** | 1.04 |
105
+
106
+ Run: `github-37351143262-1`. [Details and scope](doc/benchmark-results.md) · [JSON results](benchmarks/results/latest/index.json).
107
+ Scoped application workloads; Sentry uses historical 24.1.0. Ratios above 1 favor DBWorker; no cross-project average is computed.
108
+ <!-- benchmark-results:end -->
109
+
@@ -0,0 +1,91 @@
1
+ # dbworker
2
+
3
+ **Your database rows are already the queue.**
4
+
5
+ A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
6
+
7
+ ```text
8
+ Celery / Huey / Graphile Worker:
9
+ event → job → worker
10
+
11
+ dbworker:
12
+ row → worker
13
+ ```
14
+
15
+ Requires Python 3.12+. Install into your application:
16
+
17
+ ```sh
18
+ poetry add dbworker
19
+ ```
20
+
21
+ ## Usage
22
+
23
+ Register a handler for an existing SQLAlchemy model. DBWorker discovers eligible
24
+ rows, runs the handler in a child process, and commits its database changes with
25
+ the execution status.
26
+
27
+ ```python
28
+ from sqlalchemy import select
29
+ from sqlalchemy.orm import Session
30
+ from dbworker import Coordinator, Finished
31
+
32
+ coordinator = Coordinator(session_factory, database_url=database_url)
33
+
34
+ @coordinator.transactional_worker(
35
+ name="process",
36
+ source=YourModel,
37
+ eligible=lambda: (
38
+ select(YourModel)
39
+ .where(YourModel.your_bool_property.is_(True))
40
+ .order_by(YourModel.id)
41
+ ),
42
+ concurrency=4,
43
+ )
44
+ def process(source: YourModel, session: Session) -> Finished:
45
+ # Apply your application logic using the supplied session.
46
+ return Finished()
47
+
48
+ if __name__ == "__main__":
49
+ coordinator.create_worker_tables()
50
+ coordinator.start()
51
+ ```
52
+
53
+ `YourModel`, `session_factory`, and `database_url` come from your application.
54
+ `eligible` selects which rows can be claimed and in what order.
55
+ It is optional: omitting it selects from the whole model. DBWorker automatically
56
+ excludes completed, failed, and actively claimed work.
57
+ Return `Finished()` when processing is complete; no enqueue call is needed.
58
+
59
+ See the [Usage Instructions](https://github.com/zysilm/dbworker/blob/main/doc/usage.md) for eligibility queries, incremental
60
+ processing, service startup and shutdown, status tracking, dependencies,
61
+ failure handling, and configuration.
62
+
63
+ ## Examples
64
+
65
+ - [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
66
+ - [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
67
+ - [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
68
+
69
+ ## License
70
+
71
+ [MIT](LICENSE) © 2026 Ziyang Song.
72
+
73
+ <!-- benchmark-results:start -->
74
+ ## Benchmark Results
75
+
76
+ Median wall time in seconds; **bold** marks the faster backend. Each experiment runs in a fresh Docker container on its own GitHub-hosted Ubuntu VM.
77
+
78
+ | Experiment | Scenario | Celery (s) | DBWorker (s) | Celery / DBWorker |
79
+ |---|---|---:|---:|---:|
80
+ | imagededup | build | **4.885** | 5.939 | 0.82 |
81
+ | imagededup | comparison | 78.031 | **74.414** | 1.05 |
82
+ | imagededup | mixed | **85.559** | 86.142 | 0.99 |
83
+ | superset | sql_lab_group_by | **2.045** | 2.070 | 0.99 |
84
+ | saleor | product_csv_export | **1.912** | 2.272 | 0.84 |
85
+ | paperless_ngx | scanned_image_ingestion | 40.368 | **40.143** | 1.01 |
86
+ | posthog | pre_rendered_notification_smtp | 2.006 | **1.511** | 1.33 |
87
+ | sentry | historical_smtp_email | 157.003 | **150.927** | 1.04 |
88
+
89
+ Run: `github-37351143262-1`. [Details and scope](doc/benchmark-results.md) · [JSON results](benchmarks/results/latest/index.json).
90
+ Scoped application workloads; Sentry uses historical 24.1.0. Ratios above 1 favor DBWorker; no cross-project average is computed.
91
+ <!-- benchmark-results:end -->
@@ -4,8 +4,8 @@ build-backend = "poetry.core.masonry.api"
4
4
 
5
5
  [tool.poetry]
6
6
  name = "dbworker"
7
- version = "0.0.5"
8
- description = "Database-backed coordination for process workers."
7
+ version = "0.0.7"
8
+ description = "Portable, single-script Python/SQLAlchemy background worker for SQLite, PostgreSQL and MySQL — no Redis, broker or enqueue"
9
9
  authors = ["Ziyang Song"]
10
10
  license = "MIT"
11
11
  readme = "README.md"
dbworker-0.0.5/PKG-INFO DELETED
@@ -1,233 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: dbworker
3
- Version: 0.0.5
4
- Summary: Database-backed coordination for process workers.
5
- License: MIT
6
- License-File: LICENSE
7
- Author: Ziyang Song
8
- Requires-Python: >=3.12,<4.0
9
- Classifier: License :: OSI Approved :: MIT License
10
- Classifier: Programming Language :: Python :: 3
11
- Classifier: Programming Language :: Python :: 3.12
12
- Classifier: Programming Language :: Python :: 3.13
13
- Classifier: Programming Language :: Python :: 3.14
14
- Requires-Dist: sqlalchemy (>=2.0,<3.0)
15
- Project-URL: Repository, https://github.com/zysilm/dbworker
16
- Description-Content-Type: text/markdown
17
-
18
- # dbworker
19
-
20
- **Your database rows are already the queue.**
21
-
22
- A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
23
-
24
- ```text
25
- Celery / Huey / Graphile Worker:
26
- event → job → worker
27
-
28
- dbworker:
29
- row → worker
30
- ```
31
-
32
- Requires Python 3.12+. Install into your application:
33
-
34
- ```sh
35
- poetry add dbworker
36
- ```
37
-
38
- ## Quick start
39
-
40
- Use your existing SQLAlchemy `session_factory` and database URL:
41
-
42
- ```python
43
- from dbworker import Coordinator
44
-
45
- coordinator = Coordinator(session_factory, database_url=database_url)
46
- ```
47
-
48
- You write a **handler**: a Python function containing the work you want to run. DBWorker runs handlers in separate processes.
49
-
50
- Before calling your handler, DBWorker chooses the next task and marks it as “being worked on” in the database. This step is called a **claim**. It lets several processes share the tasks without choosing the same task at the same time.
51
-
52
- DBWorker keeps the claim active until your handler finishes. If the process crashes, the claim expires so another process can take over the task.
53
-
54
- `source` is the database table that supplies input to your handler, specified as a SQLAlchemy model. It can store user requests, or any entries whose creation should automatically start a procedure. Each new entry gives DBWorker new work to run, and the selected entry is passed to your handler. Its primary key identifies that work so DBWorker can track its completion; the model must have a single primary-key column.
55
-
56
- `eligible` controls which inputs can be claimed next. Without it, any new entry can be picked up. Add it when work should wait for a condition or run in a particular order. In the example below, `YourModel` stands for your model. The `enabled` filter and ordering by `id` illustrate a selection rule; replace them with your own conditions and ordering.
57
-
58
- Decorate your handler like this. The comments describe where your application logic goes:
59
-
60
- **Complete the work in one invocation:**
61
-
62
- ```python
63
- from sqlalchemy import select
64
- from sqlalchemy.orm import Session
65
- from dbworker import Finished
66
-
67
- @coordinator.transactional_worker(
68
- name="process",
69
- source=YourModel,
70
- eligible=lambda: (
71
- select(YourModel)
72
- .where(YourModel.enabled.is_(True))
73
- .order_by(YourModel.id)
74
- ),
75
- concurrency=4,
76
- )
77
- def process(source: YourModel, session: Session) -> Finished:
78
- # Your procedure goes here.
79
- # Return Finished() when the procedure is complete.
80
- return Finished()
81
- ```
82
-
83
- - `name` identifies the worker for status queries.
84
- - `concurrency=4` allows four handlers to run in child processes.
85
- - `source` is the selected instance of your source model.
86
- - `session` is a normal SQLAlchemy session supplied by DBWorker. The handler can leave either argument unused.
87
-
88
- Before each claim, DBWorker calls `eligible` and uses its query to select the next item. If nothing can be claimed, it waits and checks again. If your application later enables an item, a subsequent check can select it. Finished, failed, and currently claimed items are excluded automatically; you do not need to put those checks in your query. Omitting `eligible` lets DBWorker select from all entries in `source`.
89
-
90
- **Process part of the work, then continue in another invocation:**
91
-
92
- ```python
93
- from dbworker import Finished, Outcome, Unfinished
94
-
95
- @coordinator.transactional_worker(
96
- name="process_in_steps",
97
- source=YourModel,
98
- eligible=lambda: (
99
- select(YourModel)
100
- .where(YourModel.enabled.is_(True))
101
- .order_by(YourModel.id)
102
- ),
103
- concurrency=4,
104
- )
105
- def process_in_steps(source: YourModel, session: Session) -> Outcome:
106
- # Your procedure goes here.
107
- # Return Unfinished() if it needs another invocation to continue.
108
- # Return Finished() instead when it is complete.
109
- return Unfinished()
110
- ```
111
-
112
- `Finished()` means the work item is complete. `Unfinished()` means this invocation is done, but more work remains. Both save the execution status and commit any writes made through the supplied session; the handler does not have to make any database writes. After `Unfinished()`, DBWorker can invoke the handler again when eligible. Each invocation starts the function from the beginning. Your procedure decides how to continue; DBWorker does not save its position in the function. Use `Outcome` as the return annotation when a handler can return either result.
113
-
114
- If the handler raises an exception, DBWorker rolls back its transaction and marks the work failed. If your procedure has prerequisites that can be checked in the database, `eligible` can delay claiming until they are met.
115
-
116
- DBWorker owns the handler's commit and session cleanup. Do not commit or close this session yourself. For long computations, you can release a read transaction with `session.rollback()` first. Copy needed values before rollback, and do this before making writes you want to keep.
117
-
118
- ### Start and stop
119
-
120
- Register handlers at module scope in an importable module. In your worker service, initialize DBWorker's tables and start the coordinator:
121
-
122
- ```python
123
- if __name__ == "__main__":
124
- coordinator.create_worker_tables()
125
- coordinator.start()
126
- ```
127
-
128
- `start()` returns while workers keep running. When a source entry matches `eligible`, DBWorker can claim its work and call the handler—there is no enqueue call.
129
-
130
- On service shutdown, call:
131
-
132
- ```python
133
- coordinator.stop()
134
- ```
135
-
136
- `stop()` stops new claims and waits for active handlers to finish. Your API can run independently, using the same database. See the [complete example](examples/imagededup_system_dbwork/README.md) for runnable API and worker commands with shutdown handling.
137
-
138
- ## Track progress and coordinate dependent work
139
-
140
- Your application may need to show whether work is running or complete. Another worker may also depend on that information: one procedure prepares something, and a second can begin only after preparation finishes. If preparation fails, the application may need to report that failure instead of continuing.
141
-
142
- DBWorker tracks execution separately for each worker and source entry. Access it through the coordinator using the worker's registered name and the source entry's primary key; you do not need to manage or query DBWorker's internal tables yourself.
143
-
144
- Use `execution_status()` to get the current status in your API or handler:
145
-
146
- ```python
147
- from dbworker import ExecutionStatus
148
-
149
- with session_factory() as session:
150
- status = coordinator.execution_status(
151
- session, worker="process", source_id=source_id,
152
- )
153
- ```
154
-
155
- This returns `None` before the first claim, or an `ExecutionStatus` enum: `WORKING`, `UNFINISHED`, `FINISHED`, or `FAILED`.
156
-
157
- Sometimes you need to select inputs based on the status of their work—for example, list only inputs whose processing has finished. Calling `execution_status()` for each input would mean checking them individually.
158
-
159
- `has_execution_status()` lets you include that check in a database query. It returns a SQLAlchemy condition meaning: **“Does this worker have one of these execution statuses for this input?”** It does not run a query or return a Python `True` or `False` when called.
160
-
161
- ```python
162
- finished = coordinator.has_execution_status(
163
- worker="process",
164
- source_id=YourModel.id,
165
- statuses=(ExecutionStatus.FINISHED,),
166
- )
167
- ```
168
-
169
- - `worker` names the registered worker whose status you want to check.
170
- - `source_id` identifies its input. Using `YourModel.id` checks the corresponding input for each entry considered by the query.
171
- - `statuses` contains the acceptable statuses. The condition matches if any one applies; an input with no execution record does not match.
172
-
173
- Use the condition in an ordinary SQLAlchemy query:
174
-
175
- ```python
176
- query = select(YourModel).where(finished)
177
-
178
- with session_factory() as session:
179
- inputs = session.scalars(query).all()
180
- ```
181
-
182
- This returns inputs whose work under `process` is finished. The database checks their statuses as part of this query.
183
-
184
- The same condition can be combined with other query filters, including in an `eligible` query. Register the named worker before calling `has_execution_status()`.
185
-
186
- ## Understand failures and retry work
187
-
188
- When a handler raises an exception, DBWorker records the error and marks the work as `FAILED`. It will not automatically run that work again. Your application may need to show what went wrong and let someone retry after correcting the cause.
189
-
190
- `state()` reads the execution details for one input, giving you more information than its status alone:
191
-
192
- ```python
193
- with session_factory() as session:
194
- state = coordinator.workers["process"].state(session, source_id)
195
- error = state["error"] if state is not None else None
196
- ```
197
-
198
- `"process"` is the worker's registered name, and `source_id` is the input's primary key. The result is a mapping containing `execution_status`, the recorded `error`, and `lease_expires_at` (the claim's expiration time). It returns `None` if that worker has never claimed this input. You can use the error to explain the failure in your API or logs.
199
-
200
- After correcting the cause, use `reset_failed()` to allow another attempt:
201
-
202
- ```python
203
- with session_factory.begin() as session:
204
- reset = coordinator.workers["process"].reset_failed(session, source_id)
205
- ```
206
-
207
- `reset_failed()` changes a failed execution to `UNFINISHED` and clears its recorded error and claim. It returns `True` if it reset a failed execution, or `False` if there was no failed execution to reset. The `begin()` block commits this change.
208
-
209
- Resetting does not call the handler immediately. The running coordinator can claim the work again when it matches `eligible`. The handler starts from the beginning; resetting does not delete application results or progress. If the handler has effects outside the database transaction, make them safe to repeat.
210
-
211
- ## Configuration
212
-
213
- | `Coordinator` argument | Purpose | Default |
214
- |---|---|---|
215
- | `session_factory` | SQLAlchemy session factory used for coordination. | Required |
216
- | `database_url` | Database URL used by child processes; use the same database. | Required |
217
- | `engine_options` | SQLAlchemy `create_engine()` options for child processes. | `None` |
218
- | `lease_seconds` | Claim lifetime without renewal. Active claims are renewed automatically. | `30` |
219
- | `poll_seconds` | Initial wait after finding no claimable work. | `0.25` |
220
- | `max_poll_seconds` | Maximum wait after exponential backoff. Successful claims continue without waiting. | `10` |
221
-
222
- For SQLite, use a file-backed database. Worker names must start with a lowercase letter and contain only lowercase letters, digits and underscores.
223
-
224
- ## Examples
225
-
226
- - [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
227
- - [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
228
- - [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
229
-
230
- ## License
231
-
232
- [MIT](LICENSE) © 2026 Ziyang Song.
233
-
dbworker-0.0.5/README.md DELETED
@@ -1,215 +0,0 @@
1
- # dbworker
2
-
3
- **Your database rows are already the queue.**
4
-
5
- A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
6
-
7
- ```text
8
- Celery / Huey / Graphile Worker:
9
- event → job → worker
10
-
11
- dbworker:
12
- row → worker
13
- ```
14
-
15
- Requires Python 3.12+. Install into your application:
16
-
17
- ```sh
18
- poetry add dbworker
19
- ```
20
-
21
- ## Quick start
22
-
23
- Use your existing SQLAlchemy `session_factory` and database URL:
24
-
25
- ```python
26
- from dbworker import Coordinator
27
-
28
- coordinator = Coordinator(session_factory, database_url=database_url)
29
- ```
30
-
31
- You write a **handler**: a Python function containing the work you want to run. DBWorker runs handlers in separate processes.
32
-
33
- Before calling your handler, DBWorker chooses the next task and marks it as “being worked on” in the database. This step is called a **claim**. It lets several processes share the tasks without choosing the same task at the same time.
34
-
35
- DBWorker keeps the claim active until your handler finishes. If the process crashes, the claim expires so another process can take over the task.
36
-
37
- `source` is the database table that supplies input to your handler, specified as a SQLAlchemy model. It can store user requests, or any entries whose creation should automatically start a procedure. Each new entry gives DBWorker new work to run, and the selected entry is passed to your handler. Its primary key identifies that work so DBWorker can track its completion; the model must have a single primary-key column.
38
-
39
- `eligible` controls which inputs can be claimed next. Without it, any new entry can be picked up. Add it when work should wait for a condition or run in a particular order. In the example below, `YourModel` stands for your model. The `enabled` filter and ordering by `id` illustrate a selection rule; replace them with your own conditions and ordering.
40
-
41
- Decorate your handler like this. The comments describe where your application logic goes:
42
-
43
- **Complete the work in one invocation:**
44
-
45
- ```python
46
- from sqlalchemy import select
47
- from sqlalchemy.orm import Session
48
- from dbworker import Finished
49
-
50
- @coordinator.transactional_worker(
51
- name="process",
52
- source=YourModel,
53
- eligible=lambda: (
54
- select(YourModel)
55
- .where(YourModel.enabled.is_(True))
56
- .order_by(YourModel.id)
57
- ),
58
- concurrency=4,
59
- )
60
- def process(source: YourModel, session: Session) -> Finished:
61
- # Your procedure goes here.
62
- # Return Finished() when the procedure is complete.
63
- return Finished()
64
- ```
65
-
66
- - `name` identifies the worker for status queries.
67
- - `concurrency=4` allows four handlers to run in child processes.
68
- - `source` is the selected instance of your source model.
69
- - `session` is a normal SQLAlchemy session supplied by DBWorker. The handler can leave either argument unused.
70
-
71
- Before each claim, DBWorker calls `eligible` and uses its query to select the next item. If nothing can be claimed, it waits and checks again. If your application later enables an item, a subsequent check can select it. Finished, failed, and currently claimed items are excluded automatically; you do not need to put those checks in your query. Omitting `eligible` lets DBWorker select from all entries in `source`.
72
-
73
- **Process part of the work, then continue in another invocation:**
74
-
75
- ```python
76
- from dbworker import Finished, Outcome, Unfinished
77
-
78
- @coordinator.transactional_worker(
79
- name="process_in_steps",
80
- source=YourModel,
81
- eligible=lambda: (
82
- select(YourModel)
83
- .where(YourModel.enabled.is_(True))
84
- .order_by(YourModel.id)
85
- ),
86
- concurrency=4,
87
- )
88
- def process_in_steps(source: YourModel, session: Session) -> Outcome:
89
- # Your procedure goes here.
90
- # Return Unfinished() if it needs another invocation to continue.
91
- # Return Finished() instead when it is complete.
92
- return Unfinished()
93
- ```
94
-
95
- `Finished()` means the work item is complete. `Unfinished()` means this invocation is done, but more work remains. Both save the execution status and commit any writes made through the supplied session; the handler does not have to make any database writes. After `Unfinished()`, DBWorker can invoke the handler again when eligible. Each invocation starts the function from the beginning. Your procedure decides how to continue; DBWorker does not save its position in the function. Use `Outcome` as the return annotation when a handler can return either result.
96
-
97
- If the handler raises an exception, DBWorker rolls back its transaction and marks the work failed. If your procedure has prerequisites that can be checked in the database, `eligible` can delay claiming until they are met.
98
-
99
- DBWorker owns the handler's commit and session cleanup. Do not commit or close this session yourself. For long computations, you can release a read transaction with `session.rollback()` first. Copy needed values before rollback, and do this before making writes you want to keep.
100
-
101
- ### Start and stop
102
-
103
- Register handlers at module scope in an importable module. In your worker service, initialize DBWorker's tables and start the coordinator:
104
-
105
- ```python
106
- if __name__ == "__main__":
107
- coordinator.create_worker_tables()
108
- coordinator.start()
109
- ```
110
-
111
- `start()` returns while workers keep running. When a source entry matches `eligible`, DBWorker can claim its work and call the handler—there is no enqueue call.
112
-
113
- On service shutdown, call:
114
-
115
- ```python
116
- coordinator.stop()
117
- ```
118
-
119
- `stop()` stops new claims and waits for active handlers to finish. Your API can run independently, using the same database. See the [complete example](examples/imagededup_system_dbwork/README.md) for runnable API and worker commands with shutdown handling.
120
-
121
- ## Track progress and coordinate dependent work
122
-
123
- Your application may need to show whether work is running or complete. Another worker may also depend on that information: one procedure prepares something, and a second can begin only after preparation finishes. If preparation fails, the application may need to report that failure instead of continuing.
124
-
125
- DBWorker tracks execution separately for each worker and source entry. Access it through the coordinator using the worker's registered name and the source entry's primary key; you do not need to manage or query DBWorker's internal tables yourself.
126
-
127
- Use `execution_status()` to get the current status in your API or handler:
128
-
129
- ```python
130
- from dbworker import ExecutionStatus
131
-
132
- with session_factory() as session:
133
- status = coordinator.execution_status(
134
- session, worker="process", source_id=source_id,
135
- )
136
- ```
137
-
138
- This returns `None` before the first claim, or an `ExecutionStatus` enum: `WORKING`, `UNFINISHED`, `FINISHED`, or `FAILED`.
139
-
140
- Sometimes you need to select inputs based on the status of their work—for example, list only inputs whose processing has finished. Calling `execution_status()` for each input would mean checking them individually.
141
-
142
- `has_execution_status()` lets you include that check in a database query. It returns a SQLAlchemy condition meaning: **“Does this worker have one of these execution statuses for this input?”** It does not run a query or return a Python `True` or `False` when called.
143
-
144
- ```python
145
- finished = coordinator.has_execution_status(
146
- worker="process",
147
- source_id=YourModel.id,
148
- statuses=(ExecutionStatus.FINISHED,),
149
- )
150
- ```
151
-
152
- - `worker` names the registered worker whose status you want to check.
153
- - `source_id` identifies its input. Using `YourModel.id` checks the corresponding input for each entry considered by the query.
154
- - `statuses` contains the acceptable statuses. The condition matches if any one applies; an input with no execution record does not match.
155
-
156
- Use the condition in an ordinary SQLAlchemy query:
157
-
158
- ```python
159
- query = select(YourModel).where(finished)
160
-
161
- with session_factory() as session:
162
- inputs = session.scalars(query).all()
163
- ```
164
-
165
- This returns inputs whose work under `process` is finished. The database checks their statuses as part of this query.
166
-
167
- The same condition can be combined with other query filters, including in an `eligible` query. Register the named worker before calling `has_execution_status()`.
168
-
169
- ## Understand failures and retry work
170
-
171
- When a handler raises an exception, DBWorker records the error and marks the work as `FAILED`. It will not automatically run that work again. Your application may need to show what went wrong and let someone retry after correcting the cause.
172
-
173
- `state()` reads the execution details for one input, giving you more information than its status alone:
174
-
175
- ```python
176
- with session_factory() as session:
177
- state = coordinator.workers["process"].state(session, source_id)
178
- error = state["error"] if state is not None else None
179
- ```
180
-
181
- `"process"` is the worker's registered name, and `source_id` is the input's primary key. The result is a mapping containing `execution_status`, the recorded `error`, and `lease_expires_at` (the claim's expiration time). It returns `None` if that worker has never claimed this input. You can use the error to explain the failure in your API or logs.
182
-
183
- After correcting the cause, use `reset_failed()` to allow another attempt:
184
-
185
- ```python
186
- with session_factory.begin() as session:
187
- reset = coordinator.workers["process"].reset_failed(session, source_id)
188
- ```
189
-
190
- `reset_failed()` changes a failed execution to `UNFINISHED` and clears its recorded error and claim. It returns `True` if it reset a failed execution, or `False` if there was no failed execution to reset. The `begin()` block commits this change.
191
-
192
- Resetting does not call the handler immediately. The running coordinator can claim the work again when it matches `eligible`. The handler starts from the beginning; resetting does not delete application results or progress. If the handler has effects outside the database transaction, make them safe to repeat.
193
-
194
- ## Configuration
195
-
196
- | `Coordinator` argument | Purpose | Default |
197
- |---|---|---|
198
- | `session_factory` | SQLAlchemy session factory used for coordination. | Required |
199
- | `database_url` | Database URL used by child processes; use the same database. | Required |
200
- | `engine_options` | SQLAlchemy `create_engine()` options for child processes. | `None` |
201
- | `lease_seconds` | Claim lifetime without renewal. Active claims are renewed automatically. | `30` |
202
- | `poll_seconds` | Initial wait after finding no claimable work. | `0.25` |
203
- | `max_poll_seconds` | Maximum wait after exponential backoff. Successful claims continue without waiting. | `10` |
204
-
205
- For SQLite, use a file-backed database. Worker names must start with a lowercase letter and contain only lowercase letters, digits and underscores.
206
-
207
- ## Examples
208
-
209
- - [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
210
- - [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
211
- - [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
212
-
213
- ## License
214
-
215
- [MIT](LICENSE) © 2026 Ziyang Song.
File without changes
File without changes
File without changes