dbworker 0.0.6__tar.gz → 0.0.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dbworker-0.0.7/PKG-INFO +109 -0
- dbworker-0.0.7/README.md +91 -0
- {dbworker-0.0.6 → dbworker-0.0.7}/pyproject.toml +1 -1
- dbworker-0.0.6/PKG-INFO +0 -233
- dbworker-0.0.6/README.md +0 -215
- {dbworker-0.0.6 → dbworker-0.0.7}/LICENSE +0 -0
- {dbworker-0.0.6 → dbworker-0.0.7}/poetry.lock +0 -0
- {dbworker-0.0.6 → dbworker-0.0.7}/src/dbworker.py +0 -0
dbworker-0.0.7/PKG-INFO
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dbworker
|
|
3
|
+
Version: 0.0.7
|
|
4
|
+
Summary: Portable, single-script Python/SQLAlchemy background worker for SQLite, PostgreSQL and MySQL — no Redis, broker or enqueue
|
|
5
|
+
License: MIT
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Author: Ziyang Song
|
|
8
|
+
Requires-Python: >=3.12,<4.0
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
14
|
+
Requires-Dist: sqlalchemy (>=2.0,<3.0)
|
|
15
|
+
Project-URL: Repository, https://github.com/zysilm/dbworker
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# dbworker
|
|
19
|
+
|
|
20
|
+
**Your database rows are already the queue.**
|
|
21
|
+
|
|
22
|
+
A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
Celery / Huey / Graphile Worker:
|
|
26
|
+
event → job → worker
|
|
27
|
+
|
|
28
|
+
dbworker:
|
|
29
|
+
row → worker
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Requires Python 3.12+. Install into your application:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
poetry add dbworker
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Usage
|
|
39
|
+
|
|
40
|
+
Register a handler for an existing SQLAlchemy model. DBWorker discovers eligible
|
|
41
|
+
rows, runs the handler in a child process, and commits its database changes with
|
|
42
|
+
the execution status.
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
from sqlalchemy import select
|
|
46
|
+
from sqlalchemy.orm import Session
|
|
47
|
+
from dbworker import Coordinator, Finished
|
|
48
|
+
|
|
49
|
+
coordinator = Coordinator(session_factory, database_url=database_url)
|
|
50
|
+
|
|
51
|
+
@coordinator.transactional_worker(
|
|
52
|
+
name="process",
|
|
53
|
+
source=YourModel,
|
|
54
|
+
eligible=lambda: (
|
|
55
|
+
select(YourModel)
|
|
56
|
+
.where(YourModel.your_bool_property.is_(True))
|
|
57
|
+
.order_by(YourModel.id)
|
|
58
|
+
),
|
|
59
|
+
concurrency=4,
|
|
60
|
+
)
|
|
61
|
+
def process(source: YourModel, session: Session) -> Finished:
|
|
62
|
+
# Apply your application logic using the supplied session.
|
|
63
|
+
return Finished()
|
|
64
|
+
|
|
65
|
+
if __name__ == "__main__":
|
|
66
|
+
coordinator.create_worker_tables()
|
|
67
|
+
coordinator.start()
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`YourModel`, `session_factory`, and `database_url` come from your application.
|
|
71
|
+
`eligible` selects which rows can be claimed and in what order.
|
|
72
|
+
It is optional: omitting it selects from the whole model. DBWorker automatically
|
|
73
|
+
excludes completed, failed, and actively claimed work.
|
|
74
|
+
Return `Finished()` when processing is complete; no enqueue call is needed.
|
|
75
|
+
|
|
76
|
+
See the [Usage Instructions](https://github.com/zysilm/dbworker/blob/main/doc/usage.md) for eligibility queries, incremental
|
|
77
|
+
processing, service startup and shutdown, status tracking, dependencies,
|
|
78
|
+
failure handling, and configuration.
|
|
79
|
+
|
|
80
|
+
## Examples
|
|
81
|
+
|
|
82
|
+
- [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
|
|
83
|
+
- [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
|
|
84
|
+
- [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
|
|
85
|
+
|
|
86
|
+
## License
|
|
87
|
+
|
|
88
|
+
[MIT](LICENSE) © 2026 Ziyang Song.
|
|
89
|
+
|
|
90
|
+
<!-- benchmark-results:start -->
|
|
91
|
+
## Benchmark Results
|
|
92
|
+
|
|
93
|
+
Median wall time in seconds; **bold** marks the faster backend. Each experiment runs in a fresh Docker container on its own GitHub-hosted Ubuntu VM.
|
|
94
|
+
|
|
95
|
+
| Experiment | Scenario | Celery (s) | DBWorker (s) | Celery / DBWorker |
|
|
96
|
+
|---|---|---:|---:|---:|
|
|
97
|
+
| imagededup | build | **4.885** | 5.939 | 0.82 |
|
|
98
|
+
| imagededup | comparison | 78.031 | **74.414** | 1.05 |
|
|
99
|
+
| imagededup | mixed | **85.559** | 86.142 | 0.99 |
|
|
100
|
+
| superset | sql_lab_group_by | **2.045** | 2.070 | 0.99 |
|
|
101
|
+
| saleor | product_csv_export | **1.912** | 2.272 | 0.84 |
|
|
102
|
+
| paperless_ngx | scanned_image_ingestion | 40.368 | **40.143** | 1.01 |
|
|
103
|
+
| posthog | pre_rendered_notification_smtp | 2.006 | **1.511** | 1.33 |
|
|
104
|
+
| sentry | historical_smtp_email | 157.003 | **150.927** | 1.04 |
|
|
105
|
+
|
|
106
|
+
Run: `github-37351143262-1`. [Details and scope](doc/benchmark-results.md) · [JSON results](benchmarks/results/latest/index.json).
|
|
107
|
+
Scoped application workloads; Sentry uses historical 24.1.0. Ratios above 1 favor DBWorker; no cross-project average is computed.
|
|
108
|
+
<!-- benchmark-results:end -->
|
|
109
|
+
|
dbworker-0.0.7/README.md
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# dbworker
|
|
2
|
+
|
|
3
|
+
**Your database rows are already the queue.**
|
|
4
|
+
|
|
5
|
+
A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
Celery / Huey / Graphile Worker:
|
|
9
|
+
event → job → worker
|
|
10
|
+
|
|
11
|
+
dbworker:
|
|
12
|
+
row → worker
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Requires Python 3.12+. Install into your application:
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
poetry add dbworker
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Usage
|
|
22
|
+
|
|
23
|
+
Register a handler for an existing SQLAlchemy model. DBWorker discovers eligible
|
|
24
|
+
rows, runs the handler in a child process, and commits its database changes with
|
|
25
|
+
the execution status.
|
|
26
|
+
|
|
27
|
+
```python
|
|
28
|
+
from sqlalchemy import select
|
|
29
|
+
from sqlalchemy.orm import Session
|
|
30
|
+
from dbworker import Coordinator, Finished
|
|
31
|
+
|
|
32
|
+
coordinator = Coordinator(session_factory, database_url=database_url)
|
|
33
|
+
|
|
34
|
+
@coordinator.transactional_worker(
|
|
35
|
+
name="process",
|
|
36
|
+
source=YourModel,
|
|
37
|
+
eligible=lambda: (
|
|
38
|
+
select(YourModel)
|
|
39
|
+
.where(YourModel.your_bool_property.is_(True))
|
|
40
|
+
.order_by(YourModel.id)
|
|
41
|
+
),
|
|
42
|
+
concurrency=4,
|
|
43
|
+
)
|
|
44
|
+
def process(source: YourModel, session: Session) -> Finished:
|
|
45
|
+
# Apply your application logic using the supplied session.
|
|
46
|
+
return Finished()
|
|
47
|
+
|
|
48
|
+
if __name__ == "__main__":
|
|
49
|
+
coordinator.create_worker_tables()
|
|
50
|
+
coordinator.start()
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
`YourModel`, `session_factory`, and `database_url` come from your application.
|
|
54
|
+
`eligible` selects which rows can be claimed and in what order.
|
|
55
|
+
It is optional: omitting it selects from the whole model. DBWorker automatically
|
|
56
|
+
excludes completed, failed, and actively claimed work.
|
|
57
|
+
Return `Finished()` when processing is complete; no enqueue call is needed.
|
|
58
|
+
|
|
59
|
+
See the [Usage Instructions](https://github.com/zysilm/dbworker/blob/main/doc/usage.md) for eligibility queries, incremental
|
|
60
|
+
processing, service startup and shutdown, status tracking, dependencies,
|
|
61
|
+
failure handling, and configuration.
|
|
62
|
+
|
|
63
|
+
## Examples
|
|
64
|
+
|
|
65
|
+
- [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
|
|
66
|
+
- [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
|
|
67
|
+
- [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
|
|
68
|
+
|
|
69
|
+
## License
|
|
70
|
+
|
|
71
|
+
[MIT](LICENSE) © 2026 Ziyang Song.
|
|
72
|
+
|
|
73
|
+
<!-- benchmark-results:start -->
|
|
74
|
+
## Benchmark Results
|
|
75
|
+
|
|
76
|
+
Median wall time in seconds; **bold** marks the faster backend. Each experiment runs in a fresh Docker container on its own GitHub-hosted Ubuntu VM.
|
|
77
|
+
|
|
78
|
+
| Experiment | Scenario | Celery (s) | DBWorker (s) | Celery / DBWorker |
|
|
79
|
+
|---|---|---:|---:|---:|
|
|
80
|
+
| imagededup | build | **4.885** | 5.939 | 0.82 |
|
|
81
|
+
| imagededup | comparison | 78.031 | **74.414** | 1.05 |
|
|
82
|
+
| imagededup | mixed | **85.559** | 86.142 | 0.99 |
|
|
83
|
+
| superset | sql_lab_group_by | **2.045** | 2.070 | 0.99 |
|
|
84
|
+
| saleor | product_csv_export | **1.912** | 2.272 | 0.84 |
|
|
85
|
+
| paperless_ngx | scanned_image_ingestion | 40.368 | **40.143** | 1.01 |
|
|
86
|
+
| posthog | pre_rendered_notification_smtp | 2.006 | **1.511** | 1.33 |
|
|
87
|
+
| sentry | historical_smtp_email | 157.003 | **150.927** | 1.04 |
|
|
88
|
+
|
|
89
|
+
Run: `github-37351143262-1`. [Details and scope](doc/benchmark-results.md) · [JSON results](benchmarks/results/latest/index.json).
|
|
90
|
+
Scoped application workloads; Sentry uses historical 24.1.0. Ratios above 1 favor DBWorker; no cross-project average is computed.
|
|
91
|
+
<!-- benchmark-results:end -->
|
|
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
|
|
|
4
4
|
|
|
5
5
|
[tool.poetry]
|
|
6
6
|
name = "dbworker"
|
|
7
|
-
version = "0.0.
|
|
7
|
+
version = "0.0.7"
|
|
8
8
|
description = "Portable, single-script Python/SQLAlchemy background worker for SQLite, PostgreSQL and MySQL — no Redis, broker or enqueue"
|
|
9
9
|
authors = ["Ziyang Song"]
|
|
10
10
|
license = "MIT"
|
dbworker-0.0.6/PKG-INFO
DELETED
|
@@ -1,233 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: dbworker
|
|
3
|
-
Version: 0.0.6
|
|
4
|
-
Summary: Portable, single-script Python/SQLAlchemy background worker for SQLite, PostgreSQL and MySQL — no Redis, broker or enqueue
|
|
5
|
-
License: MIT
|
|
6
|
-
License-File: LICENSE
|
|
7
|
-
Author: Ziyang Song
|
|
8
|
-
Requires-Python: >=3.12,<4.0
|
|
9
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
-
Classifier: Programming Language :: Python :: 3
|
|
11
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
-
Classifier: Programming Language :: Python :: 3.14
|
|
14
|
-
Requires-Dist: sqlalchemy (>=2.0,<3.0)
|
|
15
|
-
Project-URL: Repository, https://github.com/zysilm/dbworker
|
|
16
|
-
Description-Content-Type: text/markdown
|
|
17
|
-
|
|
18
|
-
# dbworker
|
|
19
|
-
|
|
20
|
-
**Your database rows are already the queue.**
|
|
21
|
-
|
|
22
|
-
A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
|
|
23
|
-
|
|
24
|
-
```text
|
|
25
|
-
Celery / Huey / Graphile Worker:
|
|
26
|
-
event → job → worker
|
|
27
|
-
|
|
28
|
-
dbworker:
|
|
29
|
-
row → worker
|
|
30
|
-
```
|
|
31
|
-
|
|
32
|
-
Requires Python 3.12+. Install into your application:
|
|
33
|
-
|
|
34
|
-
```sh
|
|
35
|
-
poetry add dbworker
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
## Quick start
|
|
39
|
-
|
|
40
|
-
Use your existing SQLAlchemy `session_factory` and database URL:
|
|
41
|
-
|
|
42
|
-
```python
|
|
43
|
-
from dbworker import Coordinator
|
|
44
|
-
|
|
45
|
-
coordinator = Coordinator(session_factory, database_url=database_url)
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
You write a **handler**: a Python function containing the work you want to run. DBWorker runs handlers in separate processes.
|
|
49
|
-
|
|
50
|
-
Before calling your handler, DBWorker chooses the next task and marks it as “being worked on” in the database. This step is called a **claim**. It lets several processes share the tasks without choosing the same task at the same time.
|
|
51
|
-
|
|
52
|
-
DBWorker keeps the claim active until your handler finishes. If the process crashes, the claim expires so another process can take over the task.
|
|
53
|
-
|
|
54
|
-
`source` is the database table that supplies input to your handler, specified as a SQLAlchemy model. It can store user requests, or any entries whose creation should automatically start a procedure. Each new entry gives DBWorker new work to run, and the selected entry is passed to your handler. Its primary key identifies that work so DBWorker can track its completion; the model must have a single primary-key column.
|
|
55
|
-
|
|
56
|
-
`eligible` controls which inputs can be claimed next. Without it, any new entry can be picked up. Add it when work should wait for a condition or run in a particular order. In the example below, `YourModel` stands for your model. The `enabled` filter and ordering by `id` illustrate a selection rule; replace them with your own conditions and ordering.
|
|
57
|
-
|
|
58
|
-
Decorate your handler like this. The comments describe where your application logic goes:
|
|
59
|
-
|
|
60
|
-
**Complete the work in one invocation:**
|
|
61
|
-
|
|
62
|
-
```python
|
|
63
|
-
from sqlalchemy import select
|
|
64
|
-
from sqlalchemy.orm import Session
|
|
65
|
-
from dbworker import Finished
|
|
66
|
-
|
|
67
|
-
@coordinator.transactional_worker(
|
|
68
|
-
name="process",
|
|
69
|
-
source=YourModel,
|
|
70
|
-
eligible=lambda: (
|
|
71
|
-
select(YourModel)
|
|
72
|
-
.where(YourModel.enabled.is_(True))
|
|
73
|
-
.order_by(YourModel.id)
|
|
74
|
-
),
|
|
75
|
-
concurrency=4,
|
|
76
|
-
)
|
|
77
|
-
def process(source: YourModel, session: Session) -> Finished:
|
|
78
|
-
# Your procedure goes here.
|
|
79
|
-
# Return Finished() when the procedure is complete.
|
|
80
|
-
return Finished()
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
- `name` identifies the worker for status queries.
|
|
84
|
-
- `concurrency=4` allows four handlers to run in child processes.
|
|
85
|
-
- `source` is the selected instance of your source model.
|
|
86
|
-
- `session` is a normal SQLAlchemy session supplied by DBWorker. The handler can leave either argument unused.
|
|
87
|
-
|
|
88
|
-
Before each claim, DBWorker calls `eligible` and uses its query to select the next item. If nothing can be claimed, it waits and checks again. If your application later enables an item, a subsequent check can select it. Finished, failed, and currently claimed items are excluded automatically; you do not need to put those checks in your query. Omitting `eligible` lets DBWorker select from all entries in `source`.
|
|
89
|
-
|
|
90
|
-
**Process part of the work, then continue in another invocation:**
|
|
91
|
-
|
|
92
|
-
```python
|
|
93
|
-
from dbworker import Finished, Outcome, Unfinished
|
|
94
|
-
|
|
95
|
-
@coordinator.transactional_worker(
|
|
96
|
-
name="process_in_steps",
|
|
97
|
-
source=YourModel,
|
|
98
|
-
eligible=lambda: (
|
|
99
|
-
select(YourModel)
|
|
100
|
-
.where(YourModel.enabled.is_(True))
|
|
101
|
-
.order_by(YourModel.id)
|
|
102
|
-
),
|
|
103
|
-
concurrency=4,
|
|
104
|
-
)
|
|
105
|
-
def process_in_steps(source: YourModel, session: Session) -> Outcome:
|
|
106
|
-
# Your procedure goes here.
|
|
107
|
-
# Return Unfinished() if it needs another invocation to continue.
|
|
108
|
-
# Return Finished() instead when it is complete.
|
|
109
|
-
return Unfinished()
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
`Finished()` means the work item is complete. `Unfinished()` means this invocation is done, but more work remains. Both save the execution status and commit any writes made through the supplied session; the handler does not have to make any database writes. After `Unfinished()`, DBWorker can invoke the handler again when eligible. Each invocation starts the function from the beginning. Your procedure decides how to continue; DBWorker does not save its position in the function. Use `Outcome` as the return annotation when a handler can return either result.
|
|
113
|
-
|
|
114
|
-
If the handler raises an exception, DBWorker rolls back its transaction and marks the work failed. If your procedure has prerequisites that can be checked in the database, `eligible` can delay claiming until they are met.
|
|
115
|
-
|
|
116
|
-
DBWorker owns the handler's commit and session cleanup. Do not commit or close this session yourself. For long computations, you can release a read transaction with `session.rollback()` first. Copy needed values before rollback, and do this before making writes you want to keep.
|
|
117
|
-
|
|
118
|
-
### Start and stop
|
|
119
|
-
|
|
120
|
-
Register handlers at module scope in an importable module. In your worker service, initialize DBWorker's tables and start the coordinator:
|
|
121
|
-
|
|
122
|
-
```python
|
|
123
|
-
if __name__ == "__main__":
|
|
124
|
-
coordinator.create_worker_tables()
|
|
125
|
-
coordinator.start()
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
`start()` returns while workers keep running. When a source entry matches `eligible`, DBWorker can claim its work and call the handler—there is no enqueue call.
|
|
129
|
-
|
|
130
|
-
On service shutdown, call:
|
|
131
|
-
|
|
132
|
-
```python
|
|
133
|
-
coordinator.stop()
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
`stop()` stops new claims and waits for active handlers to finish. Your API can run independently, using the same database. See the [complete example](examples/imagededup_system_dbwork/README.md) for runnable API and worker commands with shutdown handling.
|
|
137
|
-
|
|
138
|
-
## Track progress and coordinate dependent work
|
|
139
|
-
|
|
140
|
-
Your application may need to show whether work is running or complete. Another worker may also depend on that information: one procedure prepares something, and a second can begin only after preparation finishes. If preparation fails, the application may need to report that failure instead of continuing.
|
|
141
|
-
|
|
142
|
-
DBWorker tracks execution separately for each worker and source entry. Access it through the coordinator using the worker's registered name and the source entry's primary key; you do not need to manage or query DBWorker's internal tables yourself.
|
|
143
|
-
|
|
144
|
-
Use `execution_status()` to get the current status in your API or handler:
|
|
145
|
-
|
|
146
|
-
```python
|
|
147
|
-
from dbworker import ExecutionStatus
|
|
148
|
-
|
|
149
|
-
with session_factory() as session:
|
|
150
|
-
status = coordinator.execution_status(
|
|
151
|
-
session, worker="process", source_id=source_id,
|
|
152
|
-
)
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
This returns `None` before the first claim, or an `ExecutionStatus` enum: `WORKING`, `UNFINISHED`, `FINISHED`, or `FAILED`.
|
|
156
|
-
|
|
157
|
-
Sometimes you need to select inputs based on the status of their work—for example, list only inputs whose processing has finished. Calling `execution_status()` for each input would mean checking them individually.
|
|
158
|
-
|
|
159
|
-
`has_execution_status()` lets you include that check in a database query. It returns a SQLAlchemy condition meaning: **“Does this worker have one of these execution statuses for this input?”** It does not run a query or return a Python `True` or `False` when called.
|
|
160
|
-
|
|
161
|
-
```python
|
|
162
|
-
finished = coordinator.has_execution_status(
|
|
163
|
-
worker="process",
|
|
164
|
-
source_id=YourModel.id,
|
|
165
|
-
statuses=(ExecutionStatus.FINISHED,),
|
|
166
|
-
)
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
- `worker` names the registered worker whose status you want to check.
|
|
170
|
-
- `source_id` identifies its input. Using `YourModel.id` checks the corresponding input for each entry considered by the query.
|
|
171
|
-
- `statuses` contains the acceptable statuses. The condition matches if any one applies; an input with no execution record does not match.
|
|
172
|
-
|
|
173
|
-
Use the condition in an ordinary SQLAlchemy query:
|
|
174
|
-
|
|
175
|
-
```python
|
|
176
|
-
query = select(YourModel).where(finished)
|
|
177
|
-
|
|
178
|
-
with session_factory() as session:
|
|
179
|
-
inputs = session.scalars(query).all()
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
This returns inputs whose work under `process` is finished. The database checks their statuses as part of this query.
|
|
183
|
-
|
|
184
|
-
The same condition can be combined with other query filters, including in an `eligible` query. Register the named worker before calling `has_execution_status()`.
|
|
185
|
-
|
|
186
|
-
## Understand failures and retry work
|
|
187
|
-
|
|
188
|
-
When a handler raises an exception, DBWorker records the error and marks the work as `FAILED`. It will not automatically run that work again. Your application may need to show what went wrong and let someone retry after correcting the cause.
|
|
189
|
-
|
|
190
|
-
`state()` reads the execution details for one input, giving you more information than its status alone:
|
|
191
|
-
|
|
192
|
-
```python
|
|
193
|
-
with session_factory() as session:
|
|
194
|
-
state = coordinator.workers["process"].state(session, source_id)
|
|
195
|
-
error = state["error"] if state is not None else None
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
`"process"` is the worker's registered name, and `source_id` is the input's primary key. The result is a mapping containing `execution_status`, the recorded `error`, and `lease_expires_at` (the claim's expiration time). It returns `None` if that worker has never claimed this input. You can use the error to explain the failure in your API or logs.
|
|
199
|
-
|
|
200
|
-
After correcting the cause, use `reset_failed()` to allow another attempt:
|
|
201
|
-
|
|
202
|
-
```python
|
|
203
|
-
with session_factory.begin() as session:
|
|
204
|
-
reset = coordinator.workers["process"].reset_failed(session, source_id)
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
`reset_failed()` changes a failed execution to `UNFINISHED` and clears its recorded error and claim. It returns `True` if it reset a failed execution, or `False` if there was no failed execution to reset. The `begin()` block commits this change.
|
|
208
|
-
|
|
209
|
-
Resetting does not call the handler immediately. The running coordinator can claim the work again when it matches `eligible`. The handler starts from the beginning; resetting does not delete application results or progress. If the handler has effects outside the database transaction, make them safe to repeat.
|
|
210
|
-
|
|
211
|
-
## Configuration
|
|
212
|
-
|
|
213
|
-
| `Coordinator` argument | Purpose | Default |
|
|
214
|
-
|---|---|---|
|
|
215
|
-
| `session_factory` | SQLAlchemy session factory used for coordination. | Required |
|
|
216
|
-
| `database_url` | Database URL used by child processes; use the same database. | Required |
|
|
217
|
-
| `engine_options` | SQLAlchemy `create_engine()` options for child processes. | `None` |
|
|
218
|
-
| `lease_seconds` | Claim lifetime without renewal. Active claims are renewed automatically. | `30` |
|
|
219
|
-
| `poll_seconds` | Initial wait after finding no claimable work. | `0.25` |
|
|
220
|
-
| `max_poll_seconds` | Maximum wait after exponential backoff. Successful claims continue without waiting. | `10` |
|
|
221
|
-
|
|
222
|
-
For SQLite, use a file-backed database. Worker names must start with a lowercase letter and contain only lowercase letters, digits and underscores.
|
|
223
|
-
|
|
224
|
-
## Examples
|
|
225
|
-
|
|
226
|
-
- [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
|
|
227
|
-
- [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
|
|
228
|
-
- [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
|
|
229
|
-
|
|
230
|
-
## License
|
|
231
|
-
|
|
232
|
-
[MIT](LICENSE) © 2026 Ziyang Song.
|
|
233
|
-
|
dbworker-0.0.6/README.md
DELETED
|
@@ -1,215 +0,0 @@
|
|
|
1
|
-
# dbworker
|
|
2
|
-
|
|
3
|
-
**Your database rows are already the queue.**
|
|
4
|
-
|
|
5
|
-
A portable, single-script background worker for SQLAlchemy. No broker, no enqueue, no duplicate job model.
|
|
6
|
-
|
|
7
|
-
```text
|
|
8
|
-
Celery / Huey / Graphile Worker:
|
|
9
|
-
event → job → worker
|
|
10
|
-
|
|
11
|
-
dbworker:
|
|
12
|
-
row → worker
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Requires Python 3.12+. Install into your application:
|
|
16
|
-
|
|
17
|
-
```sh
|
|
18
|
-
poetry add dbworker
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
## Quick start
|
|
22
|
-
|
|
23
|
-
Use your existing SQLAlchemy `session_factory` and database URL:
|
|
24
|
-
|
|
25
|
-
```python
|
|
26
|
-
from dbworker import Coordinator
|
|
27
|
-
|
|
28
|
-
coordinator = Coordinator(session_factory, database_url=database_url)
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
You write a **handler**: a Python function containing the work you want to run. DBWorker runs handlers in separate processes.
|
|
32
|
-
|
|
33
|
-
Before calling your handler, DBWorker chooses the next task and marks it as “being worked on” in the database. This step is called a **claim**. It lets several processes share the tasks without choosing the same task at the same time.
|
|
34
|
-
|
|
35
|
-
DBWorker keeps the claim active until your handler finishes. If the process crashes, the claim expires so another process can take over the task.
|
|
36
|
-
|
|
37
|
-
`source` is the database table that supplies input to your handler, specified as a SQLAlchemy model. It can store user requests, or any entries whose creation should automatically start a procedure. Each new entry gives DBWorker new work to run, and the selected entry is passed to your handler. Its primary key identifies that work so DBWorker can track its completion; the model must have a single primary-key column.
|
|
38
|
-
|
|
39
|
-
`eligible` controls which inputs can be claimed next. Without it, any new entry can be picked up. Add it when work should wait for a condition or run in a particular order. In the example below, `YourModel` stands for your model. The `enabled` filter and ordering by `id` illustrate a selection rule; replace them with your own conditions and ordering.
|
|
40
|
-
|
|
41
|
-
Decorate your handler like this. The comments describe where your application logic goes:
|
|
42
|
-
|
|
43
|
-
**Complete the work in one invocation:**
|
|
44
|
-
|
|
45
|
-
```python
|
|
46
|
-
from sqlalchemy import select
|
|
47
|
-
from sqlalchemy.orm import Session
|
|
48
|
-
from dbworker import Finished
|
|
49
|
-
|
|
50
|
-
@coordinator.transactional_worker(
|
|
51
|
-
name="process",
|
|
52
|
-
source=YourModel,
|
|
53
|
-
eligible=lambda: (
|
|
54
|
-
select(YourModel)
|
|
55
|
-
.where(YourModel.enabled.is_(True))
|
|
56
|
-
.order_by(YourModel.id)
|
|
57
|
-
),
|
|
58
|
-
concurrency=4,
|
|
59
|
-
)
|
|
60
|
-
def process(source: YourModel, session: Session) -> Finished:
|
|
61
|
-
# Your procedure goes here.
|
|
62
|
-
# Return Finished() when the procedure is complete.
|
|
63
|
-
return Finished()
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
- `name` identifies the worker for status queries.
|
|
67
|
-
- `concurrency=4` allows four handlers to run in child processes.
|
|
68
|
-
- `source` is the selected instance of your source model.
|
|
69
|
-
- `session` is a normal SQLAlchemy session supplied by DBWorker. The handler can leave either argument unused.
|
|
70
|
-
|
|
71
|
-
Before each claim, DBWorker calls `eligible` and uses its query to select the next item. If nothing can be claimed, it waits and checks again. If your application later enables an item, a subsequent check can select it. Finished, failed, and currently claimed items are excluded automatically; you do not need to put those checks in your query. Omitting `eligible` lets DBWorker select from all entries in `source`.
|
|
72
|
-
|
|
73
|
-
**Process part of the work, then continue in another invocation:**
|
|
74
|
-
|
|
75
|
-
```python
|
|
76
|
-
from dbworker import Finished, Outcome, Unfinished
|
|
77
|
-
|
|
78
|
-
@coordinator.transactional_worker(
|
|
79
|
-
name="process_in_steps",
|
|
80
|
-
source=YourModel,
|
|
81
|
-
eligible=lambda: (
|
|
82
|
-
select(YourModel)
|
|
83
|
-
.where(YourModel.enabled.is_(True))
|
|
84
|
-
.order_by(YourModel.id)
|
|
85
|
-
),
|
|
86
|
-
concurrency=4,
|
|
87
|
-
)
|
|
88
|
-
def process_in_steps(source: YourModel, session: Session) -> Outcome:
|
|
89
|
-
# Your procedure goes here.
|
|
90
|
-
# Return Unfinished() if it needs another invocation to continue.
|
|
91
|
-
# Return Finished() instead when it is complete.
|
|
92
|
-
return Unfinished()
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
`Finished()` means the work item is complete. `Unfinished()` means this invocation is done, but more work remains. Both save the execution status and commit any writes made through the supplied session; the handler does not have to make any database writes. After `Unfinished()`, DBWorker can invoke the handler again when eligible. Each invocation starts the function from the beginning. Your procedure decides how to continue; DBWorker does not save its position in the function. Use `Outcome` as the return annotation when a handler can return either result.
|
|
96
|
-
|
|
97
|
-
If the handler raises an exception, DBWorker rolls back its transaction and marks the work failed. If your procedure has prerequisites that can be checked in the database, `eligible` can delay claiming until they are met.
|
|
98
|
-
|
|
99
|
-
DBWorker owns the handler's commit and session cleanup. Do not commit or close this session yourself. For long computations, you can release a read transaction with `session.rollback()` first. Copy needed values before rollback, and do this before making writes you want to keep.
|
|
100
|
-
|
|
101
|
-
### Start and stop
|
|
102
|
-
|
|
103
|
-
Register handlers at module scope in an importable module. In your worker service, initialize DBWorker's tables and start the coordinator:
|
|
104
|
-
|
|
105
|
-
```python
|
|
106
|
-
if __name__ == "__main__":
|
|
107
|
-
coordinator.create_worker_tables()
|
|
108
|
-
coordinator.start()
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
`start()` returns while workers keep running. When a source entry matches `eligible`, DBWorker can claim its work and call the handler—there is no enqueue call.
|
|
112
|
-
|
|
113
|
-
On service shutdown, call:
|
|
114
|
-
|
|
115
|
-
```python
|
|
116
|
-
coordinator.stop()
|
|
117
|
-
```
|
|
118
|
-
|
|
119
|
-
`stop()` stops new claims and waits for active handlers to finish. Your API can run independently, using the same database. See the [complete example](examples/imagededup_system_dbwork/README.md) for runnable API and worker commands with shutdown handling.
|
|
120
|
-
|
|
121
|
-
## Track progress and coordinate dependent work
|
|
122
|
-
|
|
123
|
-
Your application may need to show whether work is running or complete. Another worker may also depend on that information: one procedure prepares something, and a second can begin only after preparation finishes. If preparation fails, the application may need to report that failure instead of continuing.
|
|
124
|
-
|
|
125
|
-
DBWorker tracks execution separately for each worker and source entry. Access it through the coordinator using the worker's registered name and the source entry's primary key; you do not need to manage or query DBWorker's internal tables yourself.
|
|
126
|
-
|
|
127
|
-
Use `execution_status()` to get the current status in your API or handler:
|
|
128
|
-
|
|
129
|
-
```python
|
|
130
|
-
from dbworker import ExecutionStatus
|
|
131
|
-
|
|
132
|
-
with session_factory() as session:
|
|
133
|
-
status = coordinator.execution_status(
|
|
134
|
-
session, worker="process", source_id=source_id,
|
|
135
|
-
)
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
This returns `None` before the first claim, or an `ExecutionStatus` enum: `WORKING`, `UNFINISHED`, `FINISHED`, or `FAILED`.
|
|
139
|
-
|
|
140
|
-
Sometimes you need to select inputs based on the status of their work—for example, list only inputs whose processing has finished. Calling `execution_status()` for each input would mean checking them individually.
|
|
141
|
-
|
|
142
|
-
`has_execution_status()` lets you include that check in a database query. It returns a SQLAlchemy condition meaning: **“Does this worker have one of these execution statuses for this input?”** It does not run a query or return a Python `True` or `False` when called.
|
|
143
|
-
|
|
144
|
-
```python
|
|
145
|
-
finished = coordinator.has_execution_status(
|
|
146
|
-
worker="process",
|
|
147
|
-
source_id=YourModel.id,
|
|
148
|
-
statuses=(ExecutionStatus.FINISHED,),
|
|
149
|
-
)
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
- `worker` names the registered worker whose status you want to check.
|
|
153
|
-
- `source_id` identifies its input. Using `YourModel.id` checks the corresponding input for each entry considered by the query.
|
|
154
|
-
- `statuses` contains the acceptable statuses. The condition matches if any one applies; an input with no execution record does not match.
|
|
155
|
-
|
|
156
|
-
Use the condition in an ordinary SQLAlchemy query:
|
|
157
|
-
|
|
158
|
-
```python
|
|
159
|
-
query = select(YourModel).where(finished)
|
|
160
|
-
|
|
161
|
-
with session_factory() as session:
|
|
162
|
-
inputs = session.scalars(query).all()
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
This returns inputs whose work under `process` is finished. The database checks their statuses as part of this query.
|
|
166
|
-
|
|
167
|
-
The same condition can be combined with other query filters, including in an `eligible` query. Register the named worker before calling `has_execution_status()`.
|
|
168
|
-
|
|
169
|
-
## Understand failures and retry work
|
|
170
|
-
|
|
171
|
-
When a handler raises an exception, DBWorker records the error and marks the work as `FAILED`. It will not automatically run that work again. Your application may need to show what went wrong and let someone retry after correcting the cause.
|
|
172
|
-
|
|
173
|
-
`state()` reads the execution details for one input, giving you more information than its status alone:
|
|
174
|
-
|
|
175
|
-
```python
|
|
176
|
-
with session_factory() as session:
|
|
177
|
-
state = coordinator.workers["process"].state(session, source_id)
|
|
178
|
-
error = state["error"] if state is not None else None
|
|
179
|
-
```
|
|
180
|
-
|
|
181
|
-
`"process"` is the worker's registered name, and `source_id` is the input's primary key. The result is a mapping containing `execution_status`, the recorded `error`, and `lease_expires_at` (the claim's expiration time). It returns `None` if that worker has never claimed this input. You can use the error to explain the failure in your API or logs.
|
|
182
|
-
|
|
183
|
-
After correcting the cause, use `reset_failed()` to allow another attempt:
|
|
184
|
-
|
|
185
|
-
```python
|
|
186
|
-
with session_factory.begin() as session:
|
|
187
|
-
reset = coordinator.workers["process"].reset_failed(session, source_id)
|
|
188
|
-
```
|
|
189
|
-
|
|
190
|
-
`reset_failed()` changes a failed execution to `UNFINISHED` and clears its recorded error and claim. It returns `True` if it reset a failed execution, or `False` if there was no failed execution to reset. The `begin()` block commits this change.
|
|
191
|
-
|
|
192
|
-
Resetting does not call the handler immediately. The running coordinator can claim the work again when it matches `eligible`. The handler starts from the beginning; resetting does not delete application results or progress. If the handler has effects outside the database transaction, make them safe to repeat.
|
|
193
|
-
|
|
194
|
-
## Configuration
|
|
195
|
-
|
|
196
|
-
| `Coordinator` argument | Purpose | Default |
|
|
197
|
-
|---|---|---|
|
|
198
|
-
| `session_factory` | SQLAlchemy session factory used for coordination. | Required |
|
|
199
|
-
| `database_url` | Database URL used by child processes; use the same database. | Required |
|
|
200
|
-
| `engine_options` | SQLAlchemy `create_engine()` options for child processes. | `None` |
|
|
201
|
-
| `lease_seconds` | Claim lifetime without renewal. Active claims are renewed automatically. | `30` |
|
|
202
|
-
| `poll_seconds` | Initial wait after finding no claimable work. | `0.25` |
|
|
203
|
-
| `max_poll_seconds` | Maximum wait after exponential backoff. Successful claims continue without waiting. | `10` |
|
|
204
|
-
|
|
205
|
-
For SQLite, use a file-backed database. Worker names must start with a lowercase letter and contain only lowercase letters, digits and underscores.
|
|
206
|
-
|
|
207
|
-
## Examples
|
|
208
|
-
|
|
209
|
-
- [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
|
|
210
|
-
- [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
|
|
211
|
-
- [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
|
|
212
|
-
|
|
213
|
-
## License
|
|
214
|
-
|
|
215
|
-
[MIT](LICENSE) © 2026 Ziyang Song.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|