localqueue 1.0.0__tar.gz → 1.0.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. {localqueue-1.0.0 → localqueue-1.0.1}/.github/workflows/ci.yml +12 -8
  2. {localqueue-1.0.0 → localqueue-1.0.1}/Cargo.lock +1 -1
  3. {localqueue-1.0.0 → localqueue-1.0.1}/Cargo.toml +1 -1
  4. localqueue-1.0.1/PKG-INFO +236 -0
  5. localqueue-1.0.1/README.md +220 -0
  6. {localqueue-1.0.0 → localqueue-1.0.1}/pyproject.toml +7 -2
  7. {localqueue-1.0.0 → localqueue-1.0.1}/src/storage.rs +25 -4
  8. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_contract.py +11 -7
  9. localqueue-1.0.0/PKG-INFO +0 -131
  10. localqueue-1.0.0/README.md +0 -118
  11. {localqueue-1.0.0 → localqueue-1.0.1}/.github/workflows/stress.yml +0 -0
  12. {localqueue-1.0.0 → localqueue-1.0.1}/.github/workflows/wheels.yml +0 -0
  13. {localqueue-1.0.0 → localqueue-1.0.1}/.gitignore +0 -0
  14. {localqueue-1.0.0 → localqueue-1.0.1}/benchmarks/README.md +0 -0
  15. {localqueue-1.0.0 → localqueue-1.0.1}/benchmarks/queue_bench.py +0 -0
  16. {localqueue-1.0.0 → localqueue-1.0.1}/docs/guarantees.md +0 -0
  17. {localqueue-1.0.0 → localqueue-1.0.1}/docs/internal/initial_idea.md +0 -0
  18. {localqueue-1.0.0 → localqueue-1.0.1}/examples/basic.py +0 -0
  19. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/__init__.py +0 -0
  20. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/core.py +0 -0
  21. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/exceptions.py +0 -0
  22. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/job.py +0 -0
  23. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/localqueue.pyi +0 -0
  24. {localqueue-1.0.0 → localqueue-1.0.1}/python/localqueue/worker.py +0 -0
  25. {localqueue-1.0.0 → localqueue-1.0.1}/src/error.rs +0 -0
  26. {localqueue-1.0.0 → localqueue-1.0.1}/src/lib.rs +0 -0
  27. {localqueue-1.0.0 → localqueue-1.0.1}/src/queue.rs +0 -0
  28. {localqueue-1.0.0 → localqueue-1.0.1}/src/schema.rs +0 -0
  29. {localqueue-1.0.0 → localqueue-1.0.1}/stress/run_soak.py +0 -0
  30. {localqueue-1.0.0 → localqueue-1.0.1}/tests/__init__.py +0 -0
  31. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_concurrency.py +0 -0
  32. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_core.py +0 -0
  33. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_gil.py +0 -0
  34. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_maintenance.py +0 -0
  35. {localqueue-1.0.0 → localqueue-1.0.1}/tests/test_worker.py +0 -0
@@ -81,14 +81,18 @@ jobs:
81
81
  python-version: ${{ matrix.python }}
82
82
  cache: pip
83
83
 
84
- - name: Install test dependencies
85
- run: python -m pip install --upgrade "maturin>=1.14,<2" "pytest>=8,<10"
86
-
87
- - name: Build the extension in the test environment
88
- run: maturin develop --release --locked
89
-
90
- - name: Run tests
91
- run: pytest -ra
84
+ - name: Build extension and run tests in a virtual environment
85
+ shell: bash
86
+ run: |
87
+ python -m venv .venv
88
+ if [ "$RUNNER_OS" = "Windows" ]; then
89
+ source .venv/Scripts/activate
90
+ else
91
+ source .venv/bin/activate
92
+ fi
93
+ python -m pip install --upgrade "maturin>=1.14,<2" "pytest>=8,<10"
94
+ maturin develop --release --locked
95
+ pytest -ra
92
96
 
93
97
  wheel-smoke:
94
98
  name: Build and smoke-test wheel
@@ -97,7 +97,7 @@ dependencies = [
97
97
 
98
98
  [[package]]
99
99
  name = "localqueue"
100
- version = "1.0.0"
100
+ version = "1.0.1"
101
101
  dependencies = [
102
102
  "pyo3",
103
103
  "rusqlite",
@@ -1,6 +1,6 @@
1
1
  [package]
2
2
  name = "localqueue"
3
- version = "1.0.0"
3
+ version = "1.0.1"
4
4
  edition = "2021"
5
5
  rust-version = "1.83"
6
6
  readme = "README.md"
@@ -0,0 +1,236 @@
1
+ Metadata-Version: 2.4
2
+ Name: localqueue
3
+ Version: 1.0.1
4
+ Requires-Dist: persist-queue==1.1.0 ; extra == 'benchmark'
5
+ Requires-Dist: pytest>=8.0 ; extra == 'dev'
6
+ Requires-Dist: maturin>=1.14 ; extra == 'dev'
7
+ Provides-Extra: benchmark
8
+ Provides-Extra: dev
9
+ Summary: A persistent, multiprocess-safe local queue for Python, backed by SQLite and Rust.
10
+ Requires-Python: >=3.10
11
+ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
12
+ Project-URL: Homepage, https://github.com/brunoportis/localqueue
13
+ Project-URL: Issues, https://github.com/brunoportis/localqueue/issues
14
+ Project-URL: Repository, https://github.com/brunoportis/localqueue
15
+
16
+ # localqueue
17
+
18
+ [![PyPI](https://img.shields.io/pypi/v/localqueue.svg)](https://pypi.org/project/localqueue/)
19
+ [![Python](https://img.shields.io/pypi/pyversions/localqueue.svg)](https://pypi.org/project/localqueue/)
20
+ [![CI](https://github.com/brunoportis/localqueue/actions/workflows/ci.yml/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/ci.yml)
21
+
22
+ A persistent, multiprocess-safe local queue for Python, backed by SQLite and Rust.
23
+
24
+ `localqueue` gives scripts and workers durable jobs with ACK/NACK, leases,
25
+ bounded retries, and dead-letter handling—without a server, daemon, or external
26
+ service.
27
+
28
+ - **Durable:** jobs survive process restarts in a local SQLite database.
29
+ - **Safe under concurrency:** multiple Python threads and processes can share a
30
+ queue on the same machine.
31
+ - **Failure-aware:** expired leases and transient failures are retried; exhausted
32
+ jobs move to a dead-letter state.
33
+ - **Fenced:** every delivery has a unique receipt, so stale workers cannot
34
+ acknowledge a newer delivery.
35
+ - **Flexible:** use multiple named queues, optional job deduplication, automatic
36
+ worker heartbeats, and custom serializers.
37
+
38
+ [Installation](#installation) · [Quick start](#quick-start) ·
39
+ [Worker](#worker) · [Guarantees](#delivery-guarantees) ·
40
+ [API](#api-overview) · [Development](#development)
41
+
42
+ ## Installation
43
+
44
+ Using [uv](https://docs.astral.sh/uv/):
45
+
46
+ ```bash
47
+ uv add localqueue
48
+ ```
49
+
50
+ Using pip:
51
+
52
+ ```bash
53
+ python -m pip install localqueue
54
+ ```
55
+
56
+ `localqueue` requires Python 3.10 or newer.
57
+
58
+ ## Quick start
59
+
60
+ ```python
61
+ from localqueue import SimpleQueue
62
+
63
+ with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
64
+ queue.put(
65
+ {"task": "send-email", "to": "hello@example.com"},
66
+ job_id="welcome-email-42",
67
+ )
68
+
69
+ job = queue.get()
70
+
71
+ try:
72
+ send_email(job.data)
73
+ except Exception as error:
74
+ queue.nack(job, last_error=str(error))
75
+ else:
76
+ queue.ack(job)
77
+ ```
78
+
79
+ The path passed to `SimpleQueue` is a directory. The queue creates and manages
80
+ `localqueue.db` inside it. Payloads are JSON-serialized by default.
81
+
82
+ ## Worker
83
+
84
+ `Worker` handles the ACK/NACK lifecycle for you. A successful handler is
85
+ acknowledged, an unexpected exception is retried, and an exception listed in
86
+ `permanent_errors` is sent directly to the dead-letter state.
87
+
88
+ ```python
89
+ from localqueue import SimpleQueue, Worker
90
+
91
+
92
+ class InvalidDeployment(Exception):
93
+ pass
94
+
95
+
96
+ def deploy(job):
97
+ print(f"Deploying {job.data['app']}@{job.data['revision']}")
98
+
99
+
100
+ with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
101
+ worker = Worker(
102
+ queue,
103
+ deploy,
104
+ permanent_errors=(InvalidDeployment,),
105
+ heartbeat_interval=10,
106
+ )
107
+ worker.run()
108
+ ```
109
+
110
+ For long-running handlers, `heartbeat_interval` renews the lease in the
111
+ background. A handler can also renew it explicitly with
112
+ `job.extend_lease(seconds)`.
113
+
114
+ ## Delivery guarantees
115
+
116
+ > [!IMPORTANT]
117
+ > `localqueue` provides **at-least-once** delivery, not exactly-once delivery.
118
+ > A worker can finish an external side effect and crash before `ack()`, causing
119
+ > the job to be delivered again. Make handlers idempotent when duplicate effects
120
+ > matter.
121
+
122
+ ```text
123
+ put() ──> ready ──> leased ──> acked
124
+
125
+ ├── nack() or expired lease ──> ready
126
+
127
+ └── fail() or retry limit ────> failed
128
+
129
+ retry_failed() ──> ready
130
+ ```
131
+
132
+ | Behavior | Guarantee |
133
+ | --- | --- |
134
+ | Successful `put()` | The job was committed to SQLite and has an internal ID. |
135
+ | Worker crash | An unacknowledged job becomes available after its lease expires. |
136
+ | Stale worker | `ack()`, `nack()`, `fail()`, and lease extension require the current receipt. |
137
+ | Retries | `max_retries` allows that many retries after the initial delivery; exhaustion moves the job to `failed`. |
138
+ | Deduplication | A `job_id` is unique within a named queue while its record exists. |
139
+ | Ordering | Ready jobs are claimed by insertion ID, but completion order is best effort under concurrency. |
140
+
141
+ By default, SQLite uses `synchronous=NORMAL`, which protects against normal
142
+ process crashes but may lose recent transactions after an operating-system or
143
+ power failure. Pass `fsync=True` to use `synchronous=FULL` for stronger
144
+ durability.
145
+
146
+ See [Delivery guarantees](https://github.com/brunoportis/localqueue/blob/main/docs/guarantees.md)
147
+ for the complete durability, lease, retry, fencing, and deduplication contract.
148
+
149
+ ## Configuration
150
+
151
+ ```python
152
+ queue = SimpleQueue(
153
+ "./data",
154
+ name="emails",
155
+ lease_seconds=60,
156
+ max_retries=3,
157
+ fsync=False,
158
+ serializer=None,
159
+ )
160
+ ```
161
+
162
+ | Option | Default | Description |
163
+ | --- | ---: | --- |
164
+ | `path` | required | Directory containing the shared `localqueue.db` file. |
165
+ | `name` | `"default"` | Logical queue name; several queues can share one database. |
166
+ | `lease_seconds` | `60.0` | Time a worker owns a delivery before it can be reclaimed. |
167
+ | `max_retries` | `3` | Retries allowed after the first delivery. |
168
+ | `fsync` | `False` | Use SQLite `synchronous=FULL` when enabled. |
169
+ | `serializer` | JSON | Object implementing `dumps(obj) -> bytes` and `loads(bytes) -> obj`. |
170
+
171
+ ## API overview
172
+
173
+ | Method | Purpose |
174
+ | --- | --- |
175
+ | `put(data, job_id=None)` | Enqueue a payload, with optional deduplication. |
176
+ | `get(block=True, timeout=None)` | Claim a job and start its lease. |
177
+ | `get_nowait()` | Claim immediately or raise `Empty`. |
178
+ | `ack(job)` | Confirm successful processing. |
179
+ | `nack(job, delay=0, last_error=None)` | Return a transient failure to the queue, optionally with a delay. |
180
+ | `fail(job, last_error=None)` | Move a permanent failure to the dead-letter state. |
181
+ | `extend_lease(job, seconds)` | Renew the current delivery lease. |
182
+ | `reclaim_expired_leases()` | Reclaim expired leases explicitly; `get()` also does this automatically. |
183
+ | `stats()` | Return `ready`, `processing`, `acked`, and `failed` counts. |
184
+ | `list_failed(limit=100, offset=0)` | Inspect dead-letter jobs. |
185
+ | `retry_failed(message_id)` | Move a dead-letter job back to `ready`. |
186
+ | `purge(older_than, include_failed=False)` | Delete old terminal records. |
187
+ | `vacuum()` | Compact the shared SQLite database. Run during a maintenance window. |
188
+
189
+ ## Architecture
190
+
191
+ The public API is a small Python facade over a native
192
+ [PyO3](https://pyo3.rs/) extension. Rust owns the transactional state machine,
193
+ while SQLite WAL mode provides local persistence and coordinates competing
194
+ writers.
195
+
196
+ ```text
197
+ Python application
198
+
199
+
200
+ SimpleQueue / Worker Python API and serialization
201
+
202
+
203
+ Rust native extension leases, receipts, retries, transactions
204
+
205
+
206
+ localqueue.db SQLite WAL, one machine
207
+ ```
208
+
209
+ All queue state lives in one SQLite table. Claims and multi-step transitions
210
+ use immediate transactions so concurrent processes cannot reserve the same
211
+ delivery. This is local infrastructure for a shared database on one machine,
212
+ not a distributed message broker.
213
+
214
+ ## Development
215
+
216
+ Build the extension in a local virtual environment and run the test suite:
217
+
218
+ ```bash
219
+ uv sync --extra dev
220
+ uv run maturin develop
221
+ uv run pytest
222
+ ```
223
+
224
+ Rust quality checks:
225
+
226
+ ```bash
227
+ cargo fmt --all --check
228
+ cargo clippy --locked --all-targets --all-features -- -D warnings
229
+ ```
230
+
231
+ Build a release wheel:
232
+
233
+ ```bash
234
+ uv run maturin build --release --locked
235
+ ```
236
+
@@ -0,0 +1,220 @@
1
+ # localqueue
2
+
3
+ [![PyPI](https://img.shields.io/pypi/v/localqueue.svg)](https://pypi.org/project/localqueue/)
4
+ [![Python](https://img.shields.io/pypi/pyversions/localqueue.svg)](https://pypi.org/project/localqueue/)
5
+ [![CI](https://github.com/brunoportis/localqueue/actions/workflows/ci.yml/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/ci.yml)
6
+
7
+ A persistent, multiprocess-safe local queue for Python, backed by SQLite and Rust.
8
+
9
+ `localqueue` gives scripts and workers durable jobs with ACK/NACK, leases,
10
+ bounded retries, and dead-letter handling—without a server, daemon, or external
11
+ service.
12
+
13
+ - **Durable:** jobs survive process restarts in a local SQLite database.
14
+ - **Safe under concurrency:** multiple Python threads and processes can share a
15
+ queue on the same machine.
16
+ - **Failure-aware:** expired leases and transient failures are retried; exhausted
17
+ jobs move to a dead-letter state.
18
+ - **Fenced:** every delivery has a unique receipt, so stale workers cannot
19
+ acknowledge a newer delivery.
20
+ - **Flexible:** use multiple named queues, optional job deduplication, automatic
21
+ worker heartbeats, and custom serializers.
22
+
23
+ [Installation](#installation) · [Quick start](#quick-start) ·
24
+ [Worker](#worker) · [Guarantees](#delivery-guarantees) ·
25
+ [API](#api-overview) · [Development](#development)
26
+
27
+ ## Installation
28
+
29
+ Using [uv](https://docs.astral.sh/uv/):
30
+
31
+ ```bash
32
+ uv add localqueue
33
+ ```
34
+
35
+ Using pip:
36
+
37
+ ```bash
38
+ python -m pip install localqueue
39
+ ```
40
+
41
+ `localqueue` requires Python 3.10 or newer.
42
+
43
+ ## Quick start
44
+
45
+ ```python
46
+ from localqueue import SimpleQueue
47
+
48
+ with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
49
+ queue.put(
50
+ {"task": "send-email", "to": "hello@example.com"},
51
+ job_id="welcome-email-42",
52
+ )
53
+
54
+ job = queue.get()
55
+
56
+ try:
57
+ send_email(job.data)
58
+ except Exception as error:
59
+ queue.nack(job, last_error=str(error))
60
+ else:
61
+ queue.ack(job)
62
+ ```
63
+
64
+ The path passed to `SimpleQueue` is a directory. The queue creates and manages
65
+ `localqueue.db` inside it. Payloads are JSON-serialized by default.
66
+
67
+ ## Worker
68
+
69
+ `Worker` handles the ACK/NACK lifecycle for you. A successful handler is
70
+ acknowledged, an unexpected exception is retried, and an exception listed in
71
+ `permanent_errors` is sent directly to the dead-letter state.
72
+
73
+ ```python
74
+ from localqueue import SimpleQueue, Worker
75
+
76
+
77
+ class InvalidDeployment(Exception):
78
+ pass
79
+
80
+
81
+ def deploy(job):
82
+ print(f"Deploying {job.data['app']}@{job.data['revision']}")
83
+
84
+
85
+ with SimpleQueue("./data", lease_seconds=30, max_retries=3) as queue:
86
+ worker = Worker(
87
+ queue,
88
+ deploy,
89
+ permanent_errors=(InvalidDeployment,),
90
+ heartbeat_interval=10,
91
+ )
92
+ worker.run()
93
+ ```
94
+
95
+ For long-running handlers, `heartbeat_interval` renews the lease in the
96
+ background. A handler can also renew it explicitly with
97
+ `job.extend_lease(seconds)`.
98
+
99
+ ## Delivery guarantees
100
+
101
+ > [!IMPORTANT]
102
+ > `localqueue` provides **at-least-once** delivery, not exactly-once delivery.
103
+ > A worker can finish an external side effect and crash before `ack()`, causing
104
+ > the job to be delivered again. Make handlers idempotent when duplicate effects
105
+ > matter.
106
+
107
+ ```text
108
+ put() ──> ready ──> leased ──> acked
109
+
110
+ ├── nack() or expired lease ──> ready
111
+
112
+ └── fail() or retry limit ────> failed
113
+
114
+ retry_failed() ──> ready
115
+ ```
116
+
117
+ | Behavior | Guarantee |
118
+ | --- | --- |
119
+ | Successful `put()` | The job was committed to SQLite and has an internal ID. |
120
+ | Worker crash | An unacknowledged job becomes available after its lease expires. |
121
+ | Stale worker | `ack()`, `nack()`, `fail()`, and lease extension require the current receipt. |
122
+ | Retries | `max_retries` allows that many retries after the initial delivery; exhaustion moves the job to `failed`. |
123
+ | Deduplication | A `job_id` is unique within a named queue while its record exists. |
124
+ | Ordering | Ready jobs are claimed by insertion ID, but completion order is best effort under concurrency. |
125
+
126
+ By default, SQLite uses `synchronous=NORMAL`, which protects against normal
127
+ process crashes but may lose recent transactions after an operating-system or
128
+ power failure. Pass `fsync=True` to use `synchronous=FULL` for stronger
129
+ durability.
130
+
131
+ See [Delivery guarantees](https://github.com/brunoportis/localqueue/blob/main/docs/guarantees.md)
132
+ for the complete durability, lease, retry, fencing, and deduplication contract.
133
+
134
+ ## Configuration
135
+
136
+ ```python
137
+ queue = SimpleQueue(
138
+ "./data",
139
+ name="emails",
140
+ lease_seconds=60,
141
+ max_retries=3,
142
+ fsync=False,
143
+ serializer=None,
144
+ )
145
+ ```
146
+
147
+ | Option | Default | Description |
148
+ | --- | ---: | --- |
149
+ | `path` | required | Directory containing the shared `localqueue.db` file. |
150
+ | `name` | `"default"` | Logical queue name; several queues can share one database. |
151
+ | `lease_seconds` | `60.0` | Time a worker owns a delivery before it can be reclaimed. |
152
+ | `max_retries` | `3` | Retries allowed after the first delivery. |
153
+ | `fsync` | `False` | Use SQLite `synchronous=FULL` when enabled. |
154
+ | `serializer` | JSON | Object implementing `dumps(obj) -> bytes` and `loads(bytes) -> obj`. |
155
+
156
+ ## API overview
157
+
158
+ | Method | Purpose |
159
+ | --- | --- |
160
+ | `put(data, job_id=None)` | Enqueue a payload, with optional deduplication. |
161
+ | `get(block=True, timeout=None)` | Claim a job and start its lease. |
162
+ | `get_nowait()` | Claim immediately or raise `Empty`. |
163
+ | `ack(job)` | Confirm successful processing. |
164
+ | `nack(job, delay=0, last_error=None)` | Return a transient failure to the queue, optionally with a delay. |
165
+ | `fail(job, last_error=None)` | Move a permanent failure to the dead-letter state. |
166
+ | `extend_lease(job, seconds)` | Renew the current delivery lease. |
167
+ | `reclaim_expired_leases()` | Reclaim expired leases explicitly; `get()` also does this automatically. |
168
+ | `stats()` | Return `ready`, `processing`, `acked`, and `failed` counts. |
169
+ | `list_failed(limit=100, offset=0)` | Inspect dead-letter jobs. |
170
+ | `retry_failed(message_id)` | Move a dead-letter job back to `ready`. |
171
+ | `purge(older_than, include_failed=False)` | Delete old terminal records. |
172
+ | `vacuum()` | Compact the shared SQLite database. Run during a maintenance window. |
173
+
174
+ ## Architecture
175
+
176
+ The public API is a small Python facade over a native
177
+ [PyO3](https://pyo3.rs/) extension. Rust owns the transactional state machine,
178
+ while SQLite WAL mode provides local persistence and coordinates competing
179
+ writers.
180
+
181
+ ```text
182
+ Python application
183
+
184
+
185
+ SimpleQueue / Worker Python API and serialization
186
+
187
+
188
+ Rust native extension leases, receipts, retries, transactions
189
+
190
+
191
+ localqueue.db SQLite WAL, one machine
192
+ ```
193
+
194
+ All queue state lives in one SQLite table. Claims and multi-step transitions
195
+ use immediate transactions so concurrent processes cannot reserve the same
196
+ delivery. This is local infrastructure for a shared database on one machine,
197
+ not a distributed message broker.
198
+
199
+ ## Development
200
+
201
+ Build the extension in a local virtual environment and run the test suite:
202
+
203
+ ```bash
204
+ uv sync --extra dev
205
+ uv run maturin develop
206
+ uv run pytest
207
+ ```
208
+
209
+ Rust quality checks:
210
+
211
+ ```bash
212
+ cargo fmt --all --check
213
+ cargo clippy --locked --all-targets --all-features -- -D warnings
214
+ ```
215
+
216
+ Build a release wheel:
217
+
218
+ ```bash
219
+ uv run maturin build --release --locked
220
+ ```
@@ -1,11 +1,16 @@
1
1
  [project]
2
2
  name = "localqueue"
3
- version = "1.0.0"
4
- description = "Fila persistente local em SQLite com ACK, lease e retry."
3
+ version = "1.0.1"
4
+ description = "A persistent, multiprocess-safe local queue for Python, backed by SQLite and Rust."
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.10"
7
7
  dependencies = []
8
8
 
9
+ [project.urls]
10
+ Homepage = "https://github.com/brunoportis/localqueue"
11
+ Repository = "https://github.com/brunoportis/localqueue"
12
+ Issues = "https://github.com/brunoportis/localqueue/issues"
13
+
9
14
  [project.optional-dependencies]
10
15
  dev = [
11
16
  "pytest>=8.0",
@@ -1,6 +1,7 @@
1
- use rusqlite::{Connection, OpenFlags};
1
+ use rusqlite::{Connection, ErrorCode, OpenFlags};
2
2
  use std::sync::Mutex;
3
- use std::time::{SystemTime, UNIX_EPOCH};
3
+ use std::thread;
4
+ use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
4
5
 
5
6
  use crate::error::{QueueError, Result};
6
7
  use crate::schema::SCHEMA_SQL;
@@ -18,10 +19,10 @@ impl Storage {
18
19
  | OpenFlags::SQLITE_OPEN_URI,
19
20
  )?;
20
21
 
21
- conn.pragma_update(None, "journal_mode", "WAL")?;
22
+ conn.pragma_update(None, "busy_timeout", 5000)?;
23
+ enable_wal(&conn)?;
22
24
  conn.pragma_update(None, "synchronous", if fsync { "FULL" } else { "NORMAL" })?;
23
25
  conn.pragma_update(None, "foreign_keys", "ON")?;
24
- conn.pragma_update(None, "busy_timeout", 5000)?;
25
26
 
26
27
  conn.execute_batch(SCHEMA_SQL)?;
27
28
 
@@ -43,6 +44,26 @@ impl Storage {
43
44
  }
44
45
  }
45
46
 
47
+ fn enable_wal(conn: &Connection) -> Result<()> {
48
+ let deadline = Instant::now() + Duration::from_secs(5);
49
+
50
+ loop {
51
+ match conn.pragma_update(None, "journal_mode", "WAL") {
52
+ Ok(()) => return Ok(()),
53
+ Err(error)
54
+ if matches!(
55
+ error,
56
+ rusqlite::Error::SqliteFailure(ref failure, _)
57
+ if matches!(failure.code, ErrorCode::DatabaseBusy | ErrorCode::DatabaseLocked)
58
+ ) && Instant::now() < deadline =>
59
+ {
60
+ thread::sleep(Duration::from_millis(25));
61
+ }
62
+ Err(error) => return Err(error.into()),
63
+ }
64
+ }
65
+ }
66
+
46
67
  pub fn now_ms() -> i64 {
47
68
  SystemTime::now()
48
69
  .duration_since(UNIX_EPOCH)
@@ -11,6 +11,13 @@ import pytest
11
11
  from localqueue import Empty, Job, LeaseExpired, LocalQueueError, SimpleQueue
12
12
 
13
13
 
14
+ def _put_job_with_same_id(path: str, result_queue) -> None:
15
+ q = SimpleQueue(path, lease_seconds=5.0)
16
+ job_id = q.put({"task": "x"}, job_id="job-123")
17
+ result_queue.put(job_id)
18
+ q.close()
19
+
20
+
14
21
  class TestContract:
15
22
  def test_two_queues_same_db(self, tmp_path):
16
23
  """Duas filas com nomes diferentes no mesmo banco."""
@@ -205,15 +212,12 @@ os._exit(1)
205
212
 
206
213
  path = tmp_path / "dedup"
207
214
 
208
- def put_job(result_queue):
209
- q = SimpleQueue(str(path), lease_seconds=5.0)
210
- job_id = q.put({"task": "x"}, job_id="job-123")
211
- result_queue.put(job_id)
212
- q.close()
213
-
214
215
  result_queue = multiprocessing.Queue()
215
216
  processes = [
216
- multiprocessing.Process(target=put_job, args=(result_queue,))
217
+ multiprocessing.Process(
218
+ target=_put_job_with_same_id,
219
+ args=(str(path), result_queue),
220
+ )
217
221
  for _ in range(2)
218
222
  ]
219
223
  for p in processes:
localqueue-1.0.0/PKG-INFO DELETED
@@ -1,131 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: localqueue
3
- Version: 1.0.0
4
- Requires-Dist: persist-queue==1.1.0 ; extra == 'benchmark'
5
- Requires-Dist: pytest>=8.0 ; extra == 'dev'
6
- Requires-Dist: maturin>=1.14 ; extra == 'dev'
7
- Provides-Extra: benchmark
8
- Provides-Extra: dev
9
- Summary: Fila persistente local em SQLite com ACK, lease e retry.
10
- Requires-Python: >=3.10
11
- Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
12
-
13
- # localqueue
14
-
15
- Fila persistente local em SQLite com ACK, lease e retry.
16
-
17
- O motor transacional é implementado em **Rust** (extensão nativa via pyo3/maturin),
18
- garantindo atomicidade mesmo com múltiplos processos e threads. A facade em
19
- **Python** mantém uma API simples e agradável.
20
-
21
- ## Características
22
-
23
- - Persistência em disco via SQLite (sobrevive a reinícios).
24
- - Semântica de ACK/NACK/falha com **receipt/fencing token**.
25
- - Lease/timeout: jobs não confirmados dentro do prazo voltam para a fila.
26
- - Retry com limite máximo; após isso vão para dead-letter.
27
- - Deduplicação opcional por `job_id`.
28
- - Extensão de lease (`job.extend_lease(...)`) e heartbeat no worker.
29
- - Estatísticas consistentes mesmo em multiprocessos.
30
- - Serialização JSON por padrão (legível e interoperável).
31
- - Suporte a múltiplas filas no mesmo banco (`name="foo"`, `name="bar"`).
32
-
33
- ## Arquitetura
34
-
35
- ```text
36
- localqueue/
37
- ├── src/ # Rust: schema, storage, queue, error, lib (pyo3)
38
- ├── python/localqueue/ # Python: SimpleQueue, Job, Worker, exceções
39
- ├── Cargo.toml
40
- └── pyproject.toml # build via maturin
41
- ```
42
-
43
- Toda a máquina de estados vive em uma única tabela SQLite (`messages`), com
44
- transações `BEGIN IMMEDIATE` para atomicidade. Não há camadas auxiliares de
45
- estado: cada operação (`put`, `get`, `ack`, `nack`, `fail`) é uma única
46
- transação.
47
-
48
- ## Uso rápido
49
-
50
- ```python
51
- from localqueue import SimpleQueue
52
-
53
- with SimpleQueue("./data", lease_seconds=30, max_retries=3) as q:
54
- q.put({"type": "deploy", "app": "jarvis", "revision": "abc123"})
55
-
56
- job = q.get()
57
- try:
58
- process(job.data)
59
- except Exception:
60
- q.nack(job)
61
- else:
62
- q.ack(job)
63
- ```
64
-
65
- ## Worker
66
-
67
- ```python
68
- from localqueue import SimpleQueue, Worker
69
-
70
- def deploy(job):
71
- print(f"Deploying {job.data['app']}@{job.data['revision']}")
72
- # Opcional: renova lease para jobs longos
73
- job.extend_lease(60)
74
-
75
- q = SimpleQueue("./data")
76
- worker = Worker(q, deploy, heartbeat_interval=10)
77
- worker.run()
78
- ```
79
-
80
- ## Multithreading / multiprocessos
81
-
82
- A fila é segura para uso com múltiplas threads e processos. O SQLite (em WAL)
83
- serializa escritas, mas `BEGIN IMMEDIATE` e `busy_timeout` garantem que
84
- operações concorrentes não corrompam o estado.
85
-
86
- ## Idempotência
87
-
88
- Mesmo com fencing token, um job pode ser entregue mais de uma vez (por
89
- exemplo, se o worker concluir um efeito externo e morrer antes de executar
90
- `ack`). A fila oferece semântica at-least-once, não exactly-once.
91
- Recomenda-se que os handlers sejam idempotentes — use `job_id` para
92
- deduplicação quando necessário.
93
-
94
- As garantias completas de durabilidade, leases, retries e fencing estão em
95
- [`docs/guarantees.md`](docs/guarantees.md).
96
-
97
- ## API resumida
98
-
99
- * ``put(data, job_id=None)`` – enfileira um item.
100
- * ``get(block=True, timeout=None)`` – retira um item com lease.
101
- * ``get_nowait()`` – variação não bloqueante.
102
- * ``ack(job)`` – confirma processamento.
103
- * ``nack(job, delay=0, last_error=None)`` – devolve à fila (erro transitório).
104
- * ``fail(job, last_error=None)`` – envia para dead-letter.
105
- * ``extend_lease(job, seconds)`` – renova o lease de um job.
106
- * ``reclaim_expired_leases()`` – recupera leases expirados manualmente.
107
- * ``stats()`` – retorna contagens de ready, processing, acked e failed.
108
-
109
- ## Manutenção
110
-
111
- * ``purge(older_than, include_failed=False)`` – remove mensagens antigas.
112
- * ``list_failed(limit=100, offset=0)`` – lista mensagens na dead-letter.
113
- * ``retry_failed(message_id)`` – move mensagem failed de volta para ready.
114
- * ``vacuum()`` – compacta todo o arquivo compartilhado ``localqueue.db``. Pode
115
- disputar o lock do SQLite com workers ativos, portanto prefira executá-lo em
116
- uma janela de manutenção.
117
-
118
- ## Desenvolvimento
119
-
120
- ```bash
121
- uv sync --extra dev
122
- maturin develop
123
- pytest
124
- ```
125
-
126
- ## Build para distribuição
127
-
128
- ```bash
129
- maturin build --release
130
- ```
131
-
@@ -1,118 +0,0 @@
1
- # localqueue
2
-
3
- Fila persistente local em SQLite com ACK, lease e retry.
4
-
5
- O motor transacional é implementado em **Rust** (extensão nativa via pyo3/maturin),
6
- garantindo atomicidade mesmo com múltiplos processos e threads. A facade em
7
- **Python** mantém uma API simples e agradável.
8
-
9
- ## Características
10
-
11
- - Persistência em disco via SQLite (sobrevive a reinícios).
12
- - Semântica de ACK/NACK/falha com **receipt/fencing token**.
13
- - Lease/timeout: jobs não confirmados dentro do prazo voltam para a fila.
14
- - Retry com limite máximo; após isso vão para dead-letter.
15
- - Deduplicação opcional por `job_id`.
16
- - Extensão de lease (`job.extend_lease(...)`) e heartbeat no worker.
17
- - Estatísticas consistentes mesmo em multiprocessos.
18
- - Serialização JSON por padrão (legível e interoperável).
19
- - Suporte a múltiplas filas no mesmo banco (`name="foo"`, `name="bar"`).
20
-
21
- ## Arquitetura
22
-
23
- ```text
24
- localqueue/
25
- ├── src/ # Rust: schema, storage, queue, error, lib (pyo3)
26
- ├── python/localqueue/ # Python: SimpleQueue, Job, Worker, exceções
27
- ├── Cargo.toml
28
- └── pyproject.toml # build via maturin
29
- ```
30
-
31
- Toda a máquina de estados vive em uma única tabela SQLite (`messages`), com
32
- transações `BEGIN IMMEDIATE` para atomicidade. Não há camadas auxiliares de
33
- estado: cada operação (`put`, `get`, `ack`, `nack`, `fail`) é uma única
34
- transação.
35
-
36
- ## Uso rápido
37
-
38
- ```python
39
- from localqueue import SimpleQueue
40
-
41
- with SimpleQueue("./data", lease_seconds=30, max_retries=3) as q:
42
- q.put({"type": "deploy", "app": "jarvis", "revision": "abc123"})
43
-
44
- job = q.get()
45
- try:
46
- process(job.data)
47
- except Exception:
48
- q.nack(job)
49
- else:
50
- q.ack(job)
51
- ```
52
-
53
- ## Worker
54
-
55
- ```python
56
- from localqueue import SimpleQueue, Worker
57
-
58
- def deploy(job):
59
- print(f"Deploying {job.data['app']}@{job.data['revision']}")
60
- # Opcional: renova lease para jobs longos
61
- job.extend_lease(60)
62
-
63
- q = SimpleQueue("./data")
64
- worker = Worker(q, deploy, heartbeat_interval=10)
65
- worker.run()
66
- ```
67
-
68
- ## Multithreading / multiprocessos
69
-
70
- A fila é segura para uso com múltiplas threads e processos. O SQLite (em WAL)
71
- serializa escritas, mas `BEGIN IMMEDIATE` e `busy_timeout` garantem que
72
- operações concorrentes não corrompam o estado.
73
-
74
- ## Idempotência
75
-
76
- Mesmo com fencing token, um job pode ser entregue mais de uma vez (por
77
- exemplo, se o worker concluir um efeito externo e morrer antes de executar
78
- `ack`). A fila oferece semântica at-least-once, não exactly-once.
79
- Recomenda-se que os handlers sejam idempotentes — use `job_id` para
80
- deduplicação quando necessário.
81
-
82
- As garantias completas de durabilidade, leases, retries e fencing estão em
83
- [`docs/guarantees.md`](docs/guarantees.md).
84
-
85
- ## API resumida
86
-
87
- * ``put(data, job_id=None)`` – enfileira um item.
88
- * ``get(block=True, timeout=None)`` – retira um item com lease.
89
- * ``get_nowait()`` – variação não bloqueante.
90
- * ``ack(job)`` – confirma processamento.
91
- * ``nack(job, delay=0, last_error=None)`` – devolve à fila (erro transitório).
92
- * ``fail(job, last_error=None)`` – envia para dead-letter.
93
- * ``extend_lease(job, seconds)`` – renova o lease de um job.
94
- * ``reclaim_expired_leases()`` – recupera leases expirados manualmente.
95
- * ``stats()`` – retorna contagens de ready, processing, acked e failed.
96
-
97
- ## Manutenção
98
-
99
- * ``purge(older_than, include_failed=False)`` – remove mensagens antigas.
100
- * ``list_failed(limit=100, offset=0)`` – lista mensagens na dead-letter.
101
- * ``retry_failed(message_id)`` – move mensagem failed de volta para ready.
102
- * ``vacuum()`` – compacta todo o arquivo compartilhado ``localqueue.db``. Pode
103
- disputar o lock do SQLite com workers ativos, portanto prefira executá-lo em
104
- uma janela de manutenção.
105
-
106
- ## Desenvolvimento
107
-
108
- ```bash
109
- uv sync --extra dev
110
- maturin develop
111
- pytest
112
- ```
113
-
114
- ## Build para distribuição
115
-
116
- ```bash
117
- maturin build --release
118
- ```
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes