async-background 0.7.2 → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +348 -198
- data/README.md +91 -103
- data/async-background.gemspec +3 -1
- data/lib/async/background/metrics.rb +3 -1
- data/lib/async/background/queue/schema.rb +6 -1
- data/lib/async/background/queue/sql.rb +15 -4
- data/lib/async/background/runner/schedule.rb +2 -0
- data/lib/async/background/runner.rb +17 -2
- data/lib/async/background/version.rb +1 -1
- data/lib/async/background/web/app.rb +159 -0
- data/lib/async/background/web/assets.rb +726 -0
- data/lib/async/background/web/auth.rb +24 -0
- data/lib/async/background/web/configuration.rb +172 -0
- data/lib/async/background/web/cursor.rb +58 -0
- data/lib/async/background/web/errors.rb +14 -0
- data/lib/async/background/web/event_hub.rb +71 -0
- data/lib/async/background/web/metrics_reader.rb +96 -0
- data/lib/async/background/web/request.rb +36 -0
- data/lib/async/background/web/response.rb +107 -0
- data/lib/async/background/web/router.rb +32 -0
- data/lib/async/background/web/serializer.rb +154 -0
- data/lib/async/background/web/snapshot.rb +247 -0
- data/lib/async/background/web/sql.rb +88 -0
- data/lib/async/background/web/stream.rb +82 -0
- data/lib/async/background/web.rb +52 -0
- metadata +46 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 823b13ecbf3740e79a15a9c24f0195a2d09bbc49bb189885d0738a0636cf913c
|
|
4
|
+
data.tar.gz: 4424725707108aaf8249b8ea62e500ccaa884bbc5392a4691cdcdc540ea5fda4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3e0d13f6b354efdd47fe4d93c464eb4411eb4658bd91fc9f3a2a154fd0300d72982680d98b454180c958ac93c15c3a7500df6c0ec346508c0791f6503525c042
|
|
7
|
+
data.tar.gz: 53a18f4ce6f47db152a303f456f5ba80cd3b706ceaee4e1aac35d752e8067c2e1eb807765117a77837e838376740a61c91690ac4714dd5853ae062c41cbd6c3e
|
data/CHANGELOG.md
CHANGED
|
@@ -1,251 +1,401 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 1.0.1
|
|
4
|
+
|
|
5
|
+
Dashboard security headers and a fiber-native rewrite of the SSE stream.
|
|
6
|
+
|
|
7
|
+
### Security
|
|
8
|
+
|
|
9
|
+
- HTML shell now ships a strict CSP and `X-Frame-Options: DENY`. The CSP is
|
|
10
|
+
`default-src 'none'; script-src 'self'; style-src 'self'; img-src 'self'
|
|
11
|
+
data:; connect-src 'self'; frame-ancestors 'none'; base-uri 'none';
|
|
12
|
+
form-action 'none'`.
|
|
13
|
+
- Every response carries `X-Content-Type-Options: nosniff`,
|
|
14
|
+
`Referrer-Policy: no-referrer`, and `Cross-Origin-Resource-Policy:
|
|
15
|
+
same-origin`.
|
|
16
|
+
- `401` and `404` responses are now JSON like every other error — minor
|
|
17
|
+
breaking change for clients that parsed the previous `text/plain` body.
|
|
18
|
+
- `HEAD` is accepted on every route that accepts `GET` (RFC 9110 §9.3.2).
|
|
19
|
+
`HEAD /api/stream` returns the SSE headers without opening a stream.
|
|
20
|
+
- `mount_path` validation is stricter: must start with `/`, no trailing
|
|
21
|
+
slash, no control characters, no whitespace.
|
|
22
|
+
- New `Configuration#logger` (any object responding to `#warn` / `#error`).
|
|
23
|
+
When set, auth-callable exceptions, internal `rescue StandardError`, and
|
|
24
|
+
SSE stream errors are surfaced instead of being silently swallowed.
|
|
25
|
+
|
|
26
|
+
### Architecture: fiber-native SSE
|
|
27
|
+
|
|
28
|
+
- Removed the per-process monitor `Thread.new` and per-subscription
|
|
29
|
+
`Mutex` + `ConditionVariable`. The dashboard web subsystem spawns **zero**
|
|
30
|
+
native threads of its own.
|
|
31
|
+
- `EventHub` is now a tiny mutex-guarded frame cache keyed by `data_version`.
|
|
32
|
+
No subscriptions, no monitor.
|
|
33
|
+
- `Stream#each` runs the entire poll-and-yield loop inside the per-request
|
|
34
|
+
Falcon fiber: `sleep` (fiber-aware), read `PRAGMA data_version`, yield the
|
|
35
|
+
`overview` frame when the version moves, yield a heartbeat otherwise.
|
|
36
|
+
- Each tab now polls `data_version` independently. At default 0.5 s and
|
|
37
|
+
realistic operator fan-out that's well under 50 SQLite header reads /
|
|
38
|
+
second per process. JSON rendering still happens at most once per version
|
|
39
|
+
thanks to the hub cache.
|
|
40
|
+
- Shutdown is clean: `App#close` marks the hub closed, the next poll raises
|
|
41
|
+
`ClosedError`, the loop exits. No thread to join, no `Subscription` to
|
|
42
|
+
unsubscribe.
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
## 1.0.0
|
|
49
|
+
|
|
50
|
+
First stable release. The queue execution contract from 0.7.2 (claim-token
|
|
51
|
+
CAS, lifecycle columns, barrier-based shutdown drain, per-status partial
|
|
52
|
+
indexes, versioned migrations) is now considered the public API.
|
|
53
|
+
|
|
54
|
+
### Dashboard
|
|
55
|
+
|
|
56
|
+
A read-only Rack-mountable UI under `require 'async/background/web'`:
|
|
57
|
+
vanilla HTML / CSS / JS, no framework, no npm.
|
|
58
|
+
|
|
59
|
+
- Endpoints: `GET /`, `GET /assets/{app.css,app.js}`,
|
|
60
|
+
`GET /api/{overview,executing,claimed,pending,done,failed,metrics,config,stream}`.
|
|
61
|
+
- The read path runs through `Async::Background::Web::Snapshot`, which
|
|
62
|
+
opens SQLite with `file:?mode=ro`, wraps a `Mutex` around a single shared
|
|
63
|
+
connection, and uses one read transaction per endpoint plus a TTL'd
|
|
64
|
+
overview cache (`counts_cache_ttl`, default 3 s).
|
|
65
|
+
- Distinguishes **executing** (`status='running' AND started_at IS NOT
|
|
66
|
+
NULL`) from **claimed** (`status='running' AND started_at IS NULL`).
|
|
67
|
+
- Cursor pagination for `done` / `failed` / `pending` using
|
|
68
|
+
`(finished_at, id)` / `(run_at, id)` tuples. Stable on ties.
|
|
69
|
+
- Args hidden by default (`expose_args: false`); when enabled, content
|
|
70
|
+
runs through `redact_args`. All user content rendered through
|
|
71
|
+
`textContent`, never `innerHTML`.
|
|
72
|
+
- `auth` is **mandatory**. `Configuration#validate!` rejects an
|
|
73
|
+
unconfigured `auth`. There is no permissive default — a falsey result
|
|
74
|
+
returns `401`.
|
|
75
|
+
|
|
76
|
+
### SSE transport
|
|
77
|
+
|
|
78
|
+
The dashboard uses a single long-lived `text/event-stream` connection per
|
|
79
|
+
browser tab instead of polling `/api/overview` every 2 seconds. One HTTP
|
|
80
|
+
connection per tab regardless of how long it stays open.
|
|
81
|
+
|
|
82
|
+
- `Configuration#transport` accepts `:sse` (default) or `:polling`. Anything
|
|
83
|
+
else raises `ConfigurationError`. The chosen transport is exposed at
|
|
84
|
+
`/api/config` so the client knows which path to take.
|
|
85
|
+
- Client opens `EventSource(mount_path + '/api/stream')` once; the server
|
|
86
|
+
pushes an `overview` event when `PRAGMA data_version` changes and a
|
|
87
|
+
`:keepalive` comment frame every 25 s.
|
|
88
|
+
- Server-supplied 5 s reconnect delay; each reconnect begins from a full
|
|
89
|
+
current snapshot (no event log).
|
|
90
|
+
- Asset URLs are fingerprinted and cached immutably by digest, so a
|
|
91
|
+
dashboard deploy can't leave a browser on incompatible HTML / JS / CSS.
|
|
92
|
+
|
|
93
|
+
### Server compatibility for SSE
|
|
94
|
+
|
|
95
|
+
SSE holds the response open for the lifetime of the dashboard tab.
|
|
96
|
+
|
|
97
|
+
- **Falcon** — recommended. Handles long-lived connections via fibers.
|
|
98
|
+
- **Puma** — works. Each open tab holds one worker thread for its lifetime;
|
|
99
|
+
fine for a handful of operators, problematic if many concurrent operators
|
|
100
|
+
would starve the worker pool.
|
|
101
|
+
- **Unicorn** — doesn't work. Blocking worker model can't hold long-lived
|
|
102
|
+
connections without timeouts. Stay on `:polling`.
|
|
103
|
+
|
|
104
|
+
See the picture in the README for what each server is actually holding.
|
|
105
|
+
|
|
106
|
+
### Configuration
|
|
107
|
+
|
|
108
|
+
```ruby
|
|
109
|
+
require 'async/background/web'
|
|
110
|
+
|
|
111
|
+
Async::Background::Queue::Store.prepare_dashboard!(path: '/var/lib/app/queue.db')
|
|
112
|
+
|
|
113
|
+
Async::Background::Web.configure do |c|
|
|
114
|
+
c.queue_path = '/var/lib/app/queue.db'
|
|
115
|
+
c.auth = ->(env) { env['warden'].user&.admin? }
|
|
116
|
+
c.expose_args = false
|
|
117
|
+
c.metrics_path = '/run/app/async-background.shm'
|
|
118
|
+
c.total_workers = 4
|
|
119
|
+
c.counts_cache_ttl = 3.0
|
|
120
|
+
c.poll_interval_ms = 2000
|
|
121
|
+
c.list_limit = 50
|
|
122
|
+
c.mount_path = '/admin/background'
|
|
123
|
+
c.title = 'My App background jobs'
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
run Async::Background::Web.app
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
### Dependencies
|
|
130
|
+
|
|
131
|
+
`rack` is optional. Required only when `require 'async/background/web'` is
|
|
132
|
+
loaded. Core gem and worker processes don't require it.
|
|
133
|
+
|
|
134
|
+
### Breaking changes from 0.7.x
|
|
135
|
+
|
|
136
|
+
None beyond what 0.7.2 already shipped. The 1.0 line locks the existing
|
|
137
|
+
contract:
|
|
138
|
+
|
|
139
|
+
- `Queue::Store#fetch` returns `claim_token` in the result hash.
|
|
140
|
+
- All terminal `Queue::Store` methods (`complete`, `fail`, `retry_or_fail`)
|
|
141
|
+
require the `claim_token:` kwarg and return CAS success boolean /
|
|
142
|
+
`:retried` / `:failed` / `nil`.
|
|
143
|
+
- Schema is versioned via `PRAGMA user_version`. Use
|
|
144
|
+
`Queue::Store.migrate!(path:)` to upgrade. Use
|
|
145
|
+
`Queue::Store.prepare_dashboard!(path:)` from the dashboard process to
|
|
146
|
+
add dashboard-only indexes.
|
|
147
|
+
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
|
|
151
|
+
|
|
3
152
|
## 0.7.2
|
|
4
153
|
|
|
5
|
-
|
|
6
|
-
|
|
154
|
+
Harden queue execution, retries, shutdown, and metrics. Adds schema v1,
|
|
155
|
+
optional dashboard indexes, and a faster enqueue path.
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
|
|
159
|
+
|
|
7
160
|
|
|
8
161
|
## 0.7.1
|
|
9
162
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
163
|
+
`Store` exposes three SQLite tuning knobs via `StoreOptions`, validated at
|
|
164
|
+
construction time so misconfigurations fail fast:
|
|
165
|
+
|
|
166
|
+
- `mmap` (`true` / `false`, default `true`) — memory-mapped I/O.
|
|
167
|
+
- `synchronous` (`:normal` / `:full` / `:extra`, default `:normal`) —
|
|
168
|
+
durability vs throughput.
|
|
169
|
+
- `wal_autocheckpoint` (`Integer` in `100..10_000`, default `1_000`) — WAL
|
|
170
|
+
checkpoint frequency in pages.
|
|
171
|
+
|
|
172
|
+
**Breaking change.** `Store.new(path:, mmap:)` → `Store.new(path:, options:
|
|
173
|
+
{ mmap: ... })`. The direct `mmap:` kwarg is removed in favor of the
|
|
174
|
+
unified `options:` hash. Update any call site that constructs `Store`
|
|
175
|
+
manually.
|
|
15
176
|
|
|
16
|
-
|
|
177
|
+
See [Get Started → Store tuning](docs/GET_STARTED.md#appendix-store-tuning)
|
|
178
|
+
for trade-offs.
|
|
17
179
|
|
|
18
|
-
|
|
19
|
-
|
|
180
|
+
|
|
181
|
+
|
|
182
|
+
|
|
20
183
|
|
|
21
184
|
## 0.6.2
|
|
22
185
|
|
|
23
|
-
|
|
24
|
-
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
186
|
+
Queue jobs gain a **configurable timeout** at three levels — call-site
|
|
187
|
+
`options:`, class-level `.options`, default 120 s — merged at enqueue time
|
|
188
|
+
so the runner just reads the final value from the payload:
|
|
189
|
+
|
|
190
|
+
```ruby
|
|
191
|
+
class HeavyImportJob
|
|
192
|
+
include Async::Background::Job
|
|
193
|
+
options timeout: 600
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
HeavyImportJob.perform_async(user_id, options: { timeout: 120 }) # wins
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Side effects: an `options TEXT` column in SQLite (added idempotently via
|
|
200
|
+
`ALTER TABLE … rescue nil` on existing databases), an extensible `options:`
|
|
201
|
+
hash across the entire enqueue chain, a `Job::Options` schema via
|
|
202
|
+
`Data.define` (unknown keys raise `ArgumentError`), and queue-timeout
|
|
203
|
+
failure logs now include the actual value (`"timed out after 120s"`).
|
|
204
|
+
|
|
205
|
+
|
|
206
|
+
|
|
207
|
+
|
|
45
208
|
|
|
46
209
|
## 0.6.1
|
|
47
210
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
- **
|
|
211
|
+
Two scheduler fixes and one notification fast path:
|
|
212
|
+
|
|
213
|
+
- **Cron busy-loop on overlap skip.** When a scheduled run was skipped
|
|
214
|
+
because the previous one was still active, the entry was re-pushed to the
|
|
215
|
+
heap without `reschedule`. `next_run_at` never advanced, so the next
|
|
216
|
+
iteration picked it up immediately. Skip branch now calls
|
|
217
|
+
`entry.reschedule(monotonic_now)` like the normal path.
|
|
218
|
+
- **Prepared statement reset on fetch error.** `@fetch_stmt.reset!` ran
|
|
219
|
+
after `execute` returned, so an exception inside `execute` left the
|
|
220
|
+
statement dirty and the next `fetch` could fail. Wrapped in
|
|
221
|
+
`begin / ensure`.
|
|
222
|
+
- **SocketNotifier: 1 connect per enqueue.** `notify_all` no longer
|
|
223
|
+
connects to all N worker sockets on every enqueue. Wakes a single worker
|
|
224
|
+
chosen by random offset, falls back through the ring only if the chosen
|
|
225
|
+
worker is dead. Happy path: 1 connect; worst case (all workers down): N.
|
|
226
|
+
- Pending lookup now uses a partial index
|
|
227
|
+
`idx_jobs_pending(run_at, id) WHERE status = 'pending'`. Smaller on disk,
|
|
228
|
+
cheaper to update, and matches the only query that uses it.
|
|
229
|
+
|
|
230
|
+
|
|
51
231
|
|
|
52
|
-
|
|
53
|
-
- **SocketNotifier: non-blocking enqueue with ring fallback** — `notify_all` no longer connects to all N worker sockets on every enqueue. `UNIXSocket.new` is a blocking, non-fiber-aware syscall, and notifying every worker blocked the Falcon reactor for N `connect()` calls on the hot HTTP enqueue path. Now wakes a single worker chosen by random offset, falling back through the ring only if the chosen worker is dead (`ECONNREFUSED` etc.). Happy path: 1 connect. Worst case (all workers down): N connects — same as before, but only when actually needed. Safe because the queue is shared in SQLite, not sharded per worker
|
|
54
|
-
- **SocketNotifier: cleaned up `UNAVAILABLE` error list** — removed `IO::WaitWritable` and `Errno::EAGAIN`. They implied "socket buffer full", but `write_nonblock` of a single byte to a freshly-opened connection cannot fill the kernel buffer. Listing them only misled readers
|
|
55
|
-
- **Store: partial index for pending lookup** — replaced `idx_jobs_status_run_at_id(status, run_at, id)` with partial index `idx_jobs_pending(run_at, id) WHERE status = 'pending'`. Smaller on disk, cheaper to update, and matches the only query that uses it (`fetch`). `done`/`failed`/`running` rows no longer occupy index pages
|
|
232
|
+
|
|
56
233
|
|
|
57
234
|
## 0.6.0
|
|
58
235
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
- **Socket cleanup race conditions** — `SocketWaker#cleanup_stale_socket` now validates if socket is truly stale by attempting connection
|
|
81
|
-
|
|
82
|
-
### Improvements
|
|
83
|
-
- Updated `docs/GET_STARTED.md` with new socket-based setup for Falcon
|
|
84
|
-
- Added section on web worker → background worker job enqueuing with full example
|
|
85
|
-
- Changed environment variable from `QUEUE_SOCKET_PATH` to `QUEUE_SOCKET_DIR` (directory instead of single socket path)
|
|
86
|
-
- Better error handling in `SocketWaker` and `SocketNotifier` with comprehensive `UNAVAILABLE` error list
|
|
87
|
-
- Integrated with `Async::Notification` for local wake-ups (shutdown signals)
|
|
88
|
-
|
|
89
|
-
### Technical Details
|
|
90
|
-
- **Why sockets over pipes?** Pipes require shared FDs across fork boundaries. The recommended Falcon setup calls `for_consumer!` in each child, which closes the writer, breaking the notification chain. Sockets use filesystem paths — any process can connect without inherited FDs.
|
|
91
|
-
- **Performance impact:** Adding ~80µs per enqueue for 8 workers (8 socket connections) vs ~100µs for SQLite transaction = negligible overhead
|
|
92
|
-
- **Graceful degradation:** If worker socket unavailable (`ENOENT`, `ECONNREFUSED`), producer silently skips — job still in database, will be picked up on next poll (5s max delay)
|
|
236
|
+
**Queue notification system rewritten.** The pipe-based `Notifier` is
|
|
237
|
+
replaced with a Unix-domain-socket architecture: each worker listens on its
|
|
238
|
+
own socket (`<dir>/async_bg_worker_N.sock`), producers broadcast wake-ups
|
|
239
|
+
via `SocketNotifier`. Fork-safe by design (no shared FDs), resilient to
|
|
240
|
+
restarts (stale-socket cleanup), and sub-100 µs wake-up latency
|
|
241
|
+
(30–80 µs typical).
|
|
242
|
+
|
|
243
|
+
**Why.** The pipe-based notifier was fundamentally broken in the
|
|
244
|
+
recommended multi-fork setup: `for_consumer!` closed the writer end in each
|
|
245
|
+
child, making `Client#push → notify` fail silently with `IOError`. All
|
|
246
|
+
writes hit `WRITE_DROPPED`, so the queue silently degraded to 5-second
|
|
247
|
+
polling.
|
|
248
|
+
|
|
249
|
+
**Breaking changes.** `Runner` now takes `queue_socket_dir:` instead of
|
|
250
|
+
`queue_notifier:`. `Notifier#for_producer!` / `Notifier#for_consumer!` are
|
|
251
|
+
removed. `Client#push` calls `notifier.notify_all`. Environment variable
|
|
252
|
+
`QUEUE_SOCKET_PATH` is replaced by `QUEUE_SOCKET_DIR` (a directory now).
|
|
253
|
+
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
|
|
93
257
|
|
|
94
258
|
## 0.5.1
|
|
95
259
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
- Added explanatory comments for each error type and handling strategy
|
|
260
|
+
CI infrastructure: full Docker-based integration testing (`Dockerfile.ci`,
|
|
261
|
+
`docker-compose.ci.yml`, `Gemfile.ci`) plus an end-to-end scenario test that
|
|
262
|
+
validates forked-worker behavior — normal execution, crash recovery after
|
|
263
|
+
SIGKILL, no duplicate execution under crashes, proper distribution across
|
|
264
|
+
the pool.
|
|
265
|
+
|
|
266
|
+
Also: `PRAGMA busy_timeout = 5000` on `Queue::Store` to prevent
|
|
267
|
+
`SQLITE_BUSY` under concurrent multi-process access; cleaner IO error
|
|
268
|
+
categorization in `Queue::Notifier` (`WRITE_DROPPED` vs `READ_EXHAUSTED`)
|
|
269
|
+
with explanatory comments.
|
|
270
|
+
|
|
271
|
+
|
|
272
|
+
|
|
273
|
+
|
|
111
274
|
|
|
112
275
|
## 0.5.0
|
|
113
276
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
- `Queue::Notifier` — extracted `IO_ERRORS` constant (`IO::WaitReadable`, `EOFError`, `IOError`) for cleaner `rescue` in `drain`
|
|
136
|
-
- `Queue::Store` — replaced index `idx_jobs_status_id(status, id)` with `idx_jobs_status_run_at_id(status, run_at, id)` for efficient delayed job lookups
|
|
137
|
-
- `Queue::Store` — `fetch` SQL now uses `WHERE status = 'pending' AND run_at <= ?` with `ORDER BY run_at, id` to process jobs in scheduled order
|
|
138
|
-
- Removed duplicated `monotonic_now` / `realtime_now` from `Runner` and `Store` — now provided by `Clock` module
|
|
139
|
-
- Updated documentation: README (Job module examples, Queue architecture diagram, Clock section), GET_STARTED (delayed jobs guide, Job module usage, minimal queue-only example)
|
|
277
|
+
**Delayed jobs.** Full support for scheduling jobs in the future:
|
|
278
|
+
|
|
279
|
+
```ruby
|
|
280
|
+
SomeJob.perform_in(60, *args)
|
|
281
|
+
SomeJob.perform_at(time, *args)
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
Backed by a new `run_at` column in the SQLite `jobs` table — jobs are only
|
|
285
|
+
fetched when `run_at <= now`.
|
|
286
|
+
|
|
287
|
+
**Job module.** Sidekiq-like `include Async::Background::Job` adds
|
|
288
|
+
`perform_async`, `perform_in`, `perform_at`, instance-level `#perform`, and
|
|
289
|
+
class-level `perform_now` delegation.
|
|
290
|
+
|
|
291
|
+
**Clock module.** Shared `monotonic_now` / `realtime_now` helpers extracted
|
|
292
|
+
to `Async::Background::Clock` and included by `Runner`, `Queue::Store`, and
|
|
293
|
+
`Queue::Client`.
|
|
294
|
+
|
|
295
|
+
|
|
296
|
+
|
|
297
|
+
|
|
140
298
|
|
|
141
299
|
## 0.4.5
|
|
142
300
|
|
|
143
|
-
|
|
144
|
-
|
|
301
|
+
**Fetch race condition fixed.** Wrapped `UPDATE ... RETURNING` in
|
|
302
|
+
`BEGIN IMMEDIATE` to prevent two workers from picking up the same job
|
|
303
|
+
simultaneously.
|
|
304
|
+
|
|
305
|
+
**mmap on Docker overlay2.** `overlay2` does not guarantee `write()` /
|
|
306
|
+
`mmap()` coherence, which corrupts the WAL under concurrent multi-process
|
|
307
|
+
access. mmap is now configurable via `queue_mmap: false` instead of being
|
|
308
|
+
hardcoded. Proper Docker setup with named volumes is documented in
|
|
309
|
+
[Get Started → Docker](docs/GET_STARTED.md#step-3--docker-setup).
|
|
145
310
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
311
|
+
Also: `PRAGMA optimize` on shutdown wrapped in `rescue nil`,
|
|
312
|
+
`PRAGMA incremental_vacuum` actually works now (`PRAGMA auto_vacuum =
|
|
313
|
+
INCREMENTAL` added to schema; only takes effect on new databases),
|
|
314
|
+
composite index `idx_jobs_status_id(status, id)` to eliminate a sort in
|
|
315
|
+
`fetch`. New `queue_mmap:` / `mmap:` parameters and a public
|
|
316
|
+
`attr_reader :queue_store` on `Runner`.
|
|
150
317
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
- **CRITICAL: mmap + Docker overlay2** — `overlay2` filesystem does not guarantee `write()`/`mmap()` coherence, causing SQLite WAL corruption under concurrent multi-process access. mmap is now configurable via `queue_mmap: false` instead of being hardcoded. Documented proper Docker setup with named volumes in `docs/GET_STARTED.md`
|
|
154
|
-
- **`PRAGMA optimize` on shutdown** — wrapped in `rescue nil` to prevent `SQLite3::BusyException` when another process holds the write lock during graceful shutdown
|
|
155
|
-
- **`PRAGMA incremental_vacuum` was a no-op** — added `PRAGMA auto_vacuum = INCREMENTAL` to schema. Without it, `incremental_vacuum` does nothing. Note: only takes effect on newly created databases; existing databases require a one-time `VACUUM`
|
|
318
|
+
**Breaking-ish.** `PRAGMAS` is now a frozen lambda `PRAGMAS.call(mmap_size)`
|
|
319
|
+
instead of a static string; update any direct reference.
|
|
156
320
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
- Added documentation: `README.md` (concise, with warning markers) and `docs/GET_STARTED.md` (step-by-step guide covering schedule config, Falcon integration, Docker setup, dynamic queue)
|
|
321
|
+
|
|
322
|
+
|
|
323
|
+
|
|
161
324
|
|
|
162
325
|
## 0.4.0
|
|
163
326
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
- `Queue::Store` — SQLite-backed persistent storage with WAL mode, prepared statements, and optimized pragmas
|
|
167
|
-
- `Queue::Notifier` — `IO.pipe`-based zero-cost wakeup between producer and consumer processes (no polling)
|
|
168
|
-
- `Queue::Client` — public API: `Async::Background::Queue.enqueue(JobClass, *args)`
|
|
169
|
-
- Automatic recovery of stale `running` jobs on worker restart
|
|
170
|
-
- Periodic cleanup of completed jobs (piggyback on fetch, every 5 minutes)
|
|
171
|
-
- `PRAGMA incremental_vacuum` when cleanup removes 100+ rows
|
|
172
|
-
- Worker isolation via `ISOLATION_FORKS` env variable — exclude specific workers from queue processing
|
|
173
|
-
- Custom database path via `queue_db_path` parameter
|
|
174
|
-
- Requires optional `sqlite3` gem (`~> 2.0`) — not included by default, must be added to Gemfile explicitly
|
|
175
|
-
- New Runner parameters: `queue_notifier:` and `queue_db_path:`
|
|
176
|
-
|
|
177
|
-
### Improvements
|
|
178
|
-
- Unified `monotonic_now` usage across `run_job` and `run_queue_job` (was using direct `Process.clock_gettime` call in `run_job`)
|
|
179
|
-
- `Queue::Notifier#drain` — moved `rescue` inside the loop to avoid stack unwinding on each drain cycle
|
|
327
|
+
**Dynamic job queue.** Enqueue jobs at runtime from any process (web,
|
|
328
|
+
console, rake) with automatic execution by background workers.
|
|
180
329
|
|
|
181
|
-
|
|
330
|
+
- `Queue::Store` — SQLite-backed persistent storage with WAL mode,
|
|
331
|
+
prepared statements, and optimized pragmas.
|
|
332
|
+
- `Queue::Notifier` — `IO.pipe`-based zero-cost wake-up between producer
|
|
333
|
+
and consumer processes.
|
|
334
|
+
- `Queue::Client` — public API: `Async::Background::Queue.enqueue
|
|
335
|
+
(JobClass, *args)`.
|
|
336
|
+
- Automatic recovery of stale `running` jobs on worker restart.
|
|
337
|
+
- Periodic cleanup of completed jobs (piggybacked on fetch, every 5 min);
|
|
338
|
+
`PRAGMA incremental_vacuum` when cleanup removes 100+ rows.
|
|
339
|
+
- `ISOLATION_FORKS` env var excludes specific workers from queue processing.
|
|
340
|
+
- Custom database path via `queue_db_path:` on `Runner`.
|
|
182
341
|
|
|
183
|
-
|
|
184
|
-
- Added optional metrics collection system using shared memory
|
|
185
|
-
- New `Metrics` class with worker-specific performance tracking
|
|
186
|
-
- Public API: `runner.metrics.enabled?`, `runner.metrics.values`, `Metrics.read_all()`
|
|
187
|
-
- Tracks total runs, successes, failures, timeouts, skips, active jobs, and execution times
|
|
188
|
-
- Requires optional `async-utilization` gem dependency
|
|
189
|
-
- Metrics stored in `/tmp/async-background.shm` with lock-free updates per worker
|
|
342
|
+
Requires the optional `sqlite3` gem (`~> 2.0`).
|
|
190
343
|
|
|
191
|
-
|
|
344
|
+
(The 0.6.0 socket-based architecture supersedes the pipe-based notifier
|
|
345
|
+
introduced here.)
|
|
192
346
|
|
|
193
|
-
### Improvements
|
|
194
|
-
- Micro-optimization in `wait_with_shutdown` method: use passed `task` parameter instead of `Async::Task.current` for better consistency and slight performance improvement
|
|
195
347
|
|
|
196
|
-
## 0.2.5
|
|
197
348
|
|
|
198
|
-
|
|
199
|
-
- Added graceful shutdown via signal handlers for SIGINT and SIGTERM
|
|
200
|
-
- Enhanced process lifecycle management with proper signal handling using `Signal.trap` and IO.pipe for async communication
|
|
201
|
-
- Improved robustness for production deployments and container orchestration
|
|
202
|
-
- Updated dependencies to work with latest Async 2.x API (removed deprecated `:parent` parameter usage)
|
|
349
|
+
|
|
203
350
|
|
|
204
|
-
## 0.
|
|
351
|
+
## 0.3.0
|
|
205
352
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
-
|
|
353
|
+
Optional metrics collection via shared memory. `Metrics` tracks per-worker
|
|
354
|
+
counters: `total_runs`, `total_successes`, `total_failures`,
|
|
355
|
+
`total_timeouts`, `total_skips`, `active_jobs`, plus last-run timestamp and
|
|
356
|
+
duration. Public API: `runner.metrics.enabled?`, `runner.metrics.values`,
|
|
357
|
+
`Metrics.read_all(total_workers:)`. Requires the optional
|
|
358
|
+
`async-utilization` gem; absent that, `enabled?` is `false` and `read_all`
|
|
359
|
+
returns `[]`. Default file: `/tmp/async-background.shm`.
|
|
212
360
|
|
|
213
|
-
## 0.2.2
|
|
214
361
|
|
|
215
|
-
### Bug Fixes
|
|
216
|
-
- **CRITICAL**: Removed logger parameter from Runner initialize (was unused). Fixed initialization to use Console.logger directly which now properly initializes in forked processes with correct context
|
|
217
362
|
|
|
218
|
-
|
|
363
|
+
|
|
219
364
|
|
|
220
|
-
|
|
221
|
-
- **CRITICAL**: Added missing `require 'console'` in main module. Logger was nil because Console gem was not imported, causing `undefined method 'info' for nil` errors on worker initialization
|
|
365
|
+
## 0.2.x
|
|
222
366
|
|
|
223
|
-
|
|
367
|
+
- **0.2.6** — `wait_with_shutdown` uses the passed `task` parameter
|
|
368
|
+
instead of `Async::Task.current`.
|
|
369
|
+
- **0.2.5** — Graceful shutdown via `SIGINT` / `SIGTERM` signal handlers
|
|
370
|
+
using `Signal.trap` and `IO.pipe`. Compatible with Async 2.x API
|
|
371
|
+
(removed deprecated `:parent`).
|
|
372
|
+
- **0.2.4** — Removed hardcoded version warning. Use semver pre-release
|
|
373
|
+
suffixes for unstable versions (e.g. `0.3.0.alpha1`).
|
|
374
|
+
- **0.2.2** — Removed unused `logger` parameter from `Runner#initialize`;
|
|
375
|
+
use `Console.logger` directly, which now initializes correctly in
|
|
376
|
+
forked processes.
|
|
377
|
+
- **0.2.1** — Added missing `require 'console'` in main module. Logger
|
|
378
|
+
was `nil`, causing `undefined method 'info' for nil` on worker
|
|
379
|
+
initialization.
|
|
380
|
+
- **0.2.0** — Removed hidden ActiveSupport dependency
|
|
381
|
+
(`safe_constantize` → `Object.const_get` + `NameError`). Job validation
|
|
382
|
+
now checks for `.perform_now` (class method) instead of `.perform`
|
|
383
|
+
(instance method). Fixed a race where an entry could disappear from the
|
|
384
|
+
heap during execution. Added `stop()` and `running?()` to `Runner`.
|
|
224
385
|
|
|
225
|
-
### Bug Fixes
|
|
226
|
-
- **CRITICAL**: Removed hidden ActiveSupport dependency. Replaced `safe_constantize` with `Object.const_get` + `NameError` handling
|
|
227
|
-
- **CRITICAL**: Fixed validator mismatch: now validates `.perform_now` (class method) instead of `.perform` (instance method)
|
|
228
|
-
- **CRITICAL**: Fixed race condition where entry could disappear from heap during execution. `reschedule` and `heap.push` now always execute after job processing
|
|
229
|
-
- Added full exception backtrace to error logs for production debugging
|
|
230
|
-
- Improved YAML security by removing `Symbol` from `permitted_classes`
|
|
231
|
-
- Removed Mutex from graceful shutdown (anti-pattern in Async). Boolean assignment is atomic in MRI
|
|
232
386
|
|
|
233
|
-
### Features
|
|
234
|
-
- Added optional `logger` parameter to Runner constructor for custom loggers (Rails.logger, etc.)
|
|
235
|
-
- Added `stop()` method for graceful shutdown
|
|
236
|
-
- Added `running?()` method to check scheduler status
|
|
237
387
|
|
|
238
|
-
|
|
239
|
-
- Job class validation now checks for `.perform_now` class method (was checking for `.perform` instance method)
|
|
388
|
+
|
|
240
389
|
|
|
241
390
|
## 0.1.0
|
|
242
391
|
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
-
|
|
246
|
-
-
|
|
247
|
-
-
|
|
248
|
-
-
|
|
249
|
-
-
|
|
250
|
-
-
|
|
251
|
-
-
|
|
392
|
+
Initial release.
|
|
393
|
+
|
|
394
|
+
- Single event loop with min-heap timer (`O(log N)` scheduling).
|
|
395
|
+
- Skip overlapping execution.
|
|
396
|
+
- Startup jitter to prevent thundering herd.
|
|
397
|
+
- Monotonic clock for interval jobs, wall clock for cron jobs.
|
|
398
|
+
- Deterministic worker sharding via `Zlib.crc32`.
|
|
399
|
+
- Semaphore-based concurrency control.
|
|
400
|
+
- Per-job timeout protection.
|
|
401
|
+
- Structured logging via Console.
|