async-background 0.6.0 → 0.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -0
- data/README.md +63 -110
- data/lib/async/background/job.rb +18 -6
- data/lib/async/background/queue/client.rb +30 -12
- data/lib/async/background/queue/socket_notifier.rb +10 -10
- data/lib/async/background/queue/store.rb +25 -16
- data/lib/async/background/runner.rb +14 -9
- data/lib/async/background/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f1890dfac1632afedbbb20c367d4d770c3037f6201bb6fac77396f07481df477
|
|
4
|
+
data.tar.gz: 1b435f70b2fd290bd569c0c081d19ae94250a181367ada7d3b4d157c213fd5b8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 0e9c2398241b48d0cc8fe66ed0b227cca3c93a5969e7f753c0a8dbb08a41462626d73709ae9eece1a254f8567f4a9044958e96a2658a77ab8f6b11bbdb6a1c14
|
|
7
|
+
data.tar.gz: 702016f46a3bbaa255735defcc2746baafe66a26400d2ad9525b80d89848970143998a706ceb4644519433cfcd558d9936c0caae4f968e335a2f8b8496afdec1
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,41 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.6.2
|
|
4
|
+
|
|
5
|
+
### Features
|
|
6
|
+
- **Configurable timeout for queue jobs** — queue jobs previously used a hardcoded 30-second timeout (`DEFAULT_TIMEOUT`). Now configurable via `options` hash at two levels:
|
|
7
|
+
```ruby
|
|
8
|
+
# Class-level default
|
|
9
|
+
class HeavyImportJob
|
|
10
|
+
include Async::Background::Job
|
|
11
|
+
options timeout: 600
|
|
12
|
+
|
|
13
|
+
def perform(user_id) = # ...
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
# Call-site override (wins over class-level)
|
|
17
|
+
HeavyImportJob.perform_async(user_id, options: { timeout: 120 })
|
|
18
|
+
```
|
|
19
|
+
Priority: call-site `options:` → class-level `options` → `DEFAULT_TIMEOUT` (30s). Options are merged at enqueue time so the runner simply reads the final value from the payload
|
|
20
|
+
- **`options:` hash across the entire enqueue chain** — single extensible contract from `perform_async` through `Client` down to `Store`. Currently supports `:timeout`, designed to accommodate future keys (e.g. `:retry`) without API changes
|
|
21
|
+
- **`Job::Options` schema via `Data.define`** — declares known option keys with types and defaults. Unknown keys raise `ArgumentError`, invalid types raise `TypeError`. No manual validation code
|
|
22
|
+
- **`options TEXT` column in SQLite** — stores the merged options hash as JSON. Extensible without schema changes when new options are added
|
|
23
|
+
|
|
24
|
+
### Improvements
|
|
25
|
+
- **Queue timeout logged on failure** — `run_queue_job` error log now includes actual timeout value: `"timed out after 120s"` instead of generic `"timed out"`
|
|
26
|
+
- **Idempotent schema migration** — existing databases get `ALTER TABLE jobs ADD COLUMN options TEXT` on first connection, wrapped in `rescue nil` for safe re-runs. New databases include the column in `CREATE TABLE`
|
|
27
|
+
|
|
28
|
+
## 0.6.1
|
|
29
|
+
|
|
30
|
+
### Bug Fixes
|
|
31
|
+
- **Runner: cron jobs busy-loop on overlap skip** — when a scheduled run was skipped because the previous one was still active, the entry was re-pushed to the heap without calling `reschedule`. For cron jobs (where `interval` is `nil`), this meant `next_run_at` was never advanced to the next cron tick, causing the entry to be picked up again immediately on the next loop iteration. Skip branch now calls `entry.reschedule(monotonic_now)` like the normal path
|
|
32
|
+
- **Store: prepared statement not reset on fetch error** — `@fetch_stmt.reset!` was called after `execute` returned, so an exception inside `execute` left the statement in a dirty state and the next `fetch` could fail. Wrapped in `begin/ensure` to guarantee reset
|
|
33
|
+
|
|
34
|
+
### Improvements
|
|
35
|
+
- **SocketNotifier: non-blocking enqueue with ring fallback** — `notify_all` no longer connects to all N worker sockets on every enqueue. `UNIXSocket.new` is a blocking, non-fiber-aware syscall, and notifying every worker blocked the Falcon reactor for N `connect()` calls on the hot HTTP enqueue path. Now wakes a single worker chosen by random offset, falling back through the ring only if the chosen worker is dead (`ECONNREFUSED` etc.). Happy path: 1 connect. Worst case (all workers down): N connects — same as before, but only when actually needed. Safe because the queue is shared in SQLite, not sharded per worker
|
|
36
|
+
- **SocketNotifier: cleaned up `UNAVAILABLE` error list** — removed `IO::WaitWritable` and `Errno::EAGAIN`. They implied "socket buffer full", but `write_nonblock` of a single byte to a freshly-opened connection cannot fill the kernel buffer. Listing them only misled readers
|
|
37
|
+
- **Store: partial index for pending lookup** — replaced `idx_jobs_status_run_at_id(status, run_at, id)` with partial index `idx_jobs_pending(run_at, id) WHERE status = 'pending'`. Smaller on disk, cheaper to update, and matches the only query that uses it (`fetch`). `done`/`failed`/`running` rows no longer occupy index pages
|
|
38
|
+
|
|
3
39
|
## 0.6.0
|
|
4
40
|
|
|
5
41
|
### Breaking Changes
|
data/README.md
CHANGED
|
@@ -1,76 +1,74 @@
|
|
|
1
1
|
# Async::Background
|
|
2
2
|
|
|
3
|
-
A lightweight,
|
|
3
|
+
A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https://github.com/socketry/async) ecosystem. Built for [Falcon](https://github.com/socketry/falcon), works with any Async app.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
- **
|
|
8
|
-
- **
|
|
9
|
-
- **
|
|
10
|
-
- **Multi-process safe** — deterministic worker sharding via `Zlib.crc32`, no duplicate execution
|
|
11
|
-
- **Skip overlapping** — if a job is still running when its next tick arrives, the tick is skipped
|
|
12
|
-
- **Timeout protection** — per-job configurable timeout via `Async::Task#with_timeout`
|
|
13
|
-
- **Startup jitter** — random delay to prevent thundering herd after restart
|
|
14
|
-
- **Optional metrics** — shared memory performance tracking with `async-utilization`
|
|
5
|
+
- **Cron & interval scheduling** on a single event loop with a min-heap
|
|
6
|
+
- **Dynamic job queue** backed by SQLite, with delayed jobs (`perform_in` / `perform_at`)
|
|
7
|
+
- **Cross-process wake-ups** over Unix domain sockets — web workers can enqueue and instantly wake background workers
|
|
8
|
+
- **Multi-process safe** — deterministic worker sharding, no duplicate execution
|
|
9
|
+
- **Per-job timeouts**, skip-on-overlap, startup jitter, optional metrics
|
|
15
10
|
|
|
16
11
|
## Requirements
|
|
17
12
|
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
13
|
+
- Ruby >= 3.3
|
|
14
|
+
- `async ~> 2.0`, `fugit ~> 1.0`
|
|
15
|
+
- `sqlite3 ~> 2.0` (optional, for the job queue)
|
|
16
|
+
- `async-utilization ~> 0.3` (optional, for metrics)
|
|
21
17
|
|
|
22
|
-
##
|
|
18
|
+
## Install
|
|
23
19
|
|
|
24
20
|
```ruby
|
|
25
21
|
# Gemfile
|
|
26
22
|
gem "async-background"
|
|
27
|
-
|
|
28
|
-
#
|
|
29
|
-
gem "sqlite3", "~> 2.0" # for dynamic job queue
|
|
30
|
-
gem "async-utilization", "~> 0.3" # for metrics
|
|
23
|
+
gem "sqlite3", "~> 2.0" # optional
|
|
24
|
+
gem "async-utilization", "~> 0.3" # optional
|
|
31
25
|
```
|
|
32
26
|
|
|
33
27
|
## ➡️ [Get Started](docs/GET_STARTED.md)
|
|
34
28
|
|
|
35
|
-
|
|
29
|
+
Full setup walkthrough: schedule config, Falcon integration, Docker, queue, delayed jobs.
|
|
36
30
|
|
|
37
31
|
---
|
|
38
32
|
|
|
39
|
-
## Quick
|
|
40
|
-
|
|
41
|
-
Include `Async::Background::Job` for a Sidekiq-like interface:
|
|
33
|
+
## Quick Look
|
|
42
34
|
|
|
43
35
|
```ruby
|
|
44
36
|
class SendEmailJob
|
|
45
37
|
include Async::Background::Job
|
|
46
38
|
|
|
47
39
|
def perform(user_id, template)
|
|
48
|
-
|
|
49
|
-
Mailer.send(user, template)
|
|
40
|
+
Mailer.send(User.find(user_id), template)
|
|
50
41
|
end
|
|
51
42
|
end
|
|
52
43
|
|
|
53
|
-
# Immediate execution in the queue
|
|
54
44
|
SendEmailJob.perform_async(user_id, "welcome")
|
|
55
|
-
|
|
56
|
-
# Execute after 5 minutes
|
|
57
45
|
SendEmailJob.perform_in(300, user_id, "reminder")
|
|
58
|
-
|
|
59
|
-
# Execute at a specific time
|
|
60
|
-
SendEmailJob.perform_at(Time.new(2026, 4, 1, 9, 0, 0), user_id, "scheduled")
|
|
46
|
+
SendEmailJob.perform_at(Time.new(2026, 4, 1, 9), user_id, "scheduled")
|
|
61
47
|
```
|
|
62
48
|
|
|
63
|
-
|
|
49
|
+
Schedule recurring jobs in `config/schedule.yml`:
|
|
64
50
|
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
51
|
+
```yaml
|
|
52
|
+
sync_products:
|
|
53
|
+
class: SyncProductsJob
|
|
54
|
+
every: 60
|
|
55
|
+
|
|
56
|
+
daily_report:
|
|
57
|
+
class: DailyReportJob
|
|
58
|
+
cron: "0 3 * * *"
|
|
59
|
+
timeout: 120
|
|
69
60
|
```
|
|
70
61
|
|
|
62
|
+
| Key | Description |
|
|
63
|
+
|---|---|
|
|
64
|
+
| `class` | Job class — must include `Async::Background::Job` |
|
|
65
|
+
| `every` / `cron` | One of: interval in seconds, or cron expression |
|
|
66
|
+
| `timeout` | Max execution time in seconds (default: 30) |
|
|
67
|
+
| `worker` | Pin to a specific worker. Default: `crc32(name) % total_workers` |
|
|
68
|
+
|
|
71
69
|
---
|
|
72
70
|
|
|
73
|
-
##
|
|
71
|
+
## Gotchas
|
|
74
72
|
|
|
75
73
|
### Docker: SQLite requires a named volume
|
|
76
74
|
|
|
@@ -78,18 +76,20 @@ The SQLite database **must not** live on Docker's `overlay2` filesystem. The `ov
|
|
|
78
76
|
|
|
79
77
|
```yaml
|
|
80
78
|
# docker-compose.yml
|
|
81
|
-
|
|
82
|
-
|
|
79
|
+
services:
|
|
80
|
+
app:
|
|
81
|
+
volumes:
|
|
82
|
+
- queue-data:/app/tmp/queue # ← named volume, NOT overlay2
|
|
83
83
|
|
|
84
84
|
volumes:
|
|
85
85
|
queue-data:
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
-
Without this, you will get database crashes in multi-process mode. See [Get Started → Step 3](docs/GET_STARTED.md#step-3-docker) for details.
|
|
88
|
+
Without this, you will get database crashes in multi-process mode. See [Get Started → Step 3](docs/GET_STARTED.md#step-3-docker) for details. If you can't use a named volume, pass `queue_mmap: false` to disable mmap entirely.
|
|
89
89
|
|
|
90
|
-
###
|
|
90
|
+
### Other gotchas
|
|
91
91
|
|
|
92
|
-
SQLite connections
|
|
92
|
+
**Don't share SQLite connections across `fork()`.** The gem opens connections lazily after fork, but if you create a `Queue::Store` manually for schema setup, close it before forking:
|
|
93
93
|
|
|
94
94
|
```ruby
|
|
95
95
|
store = Async::Background::Queue::Store.new(path: db_path)
|
|
@@ -97,96 +97,49 @@ store.ensure_database!
|
|
|
97
97
|
store.close # ← before fork
|
|
98
98
|
```
|
|
99
99
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
The `Clock` module provides shared time helpers used across the codebase:
|
|
103
|
-
|
|
104
|
-
- **`monotonic_now`** (`CLOCK_MONOTONIC`) — for in-process intervals and durations, immune to NTP drift / wall-clock jumps
|
|
105
|
-
- **`realtime_now`** (`CLOCK_REALTIME`) — for persisted timestamps (SQLite `run_at`, `created_at`, `locked_at`)
|
|
106
|
-
|
|
107
|
-
Interval jobs use monotonic clock. Cron jobs use `Time.now` because "every day at 3am" must respect real time. These are different clocks by design.
|
|
100
|
+
**Two clocks, on purpose.** Interval jobs use `CLOCK_MONOTONIC` (immune to NTP drift). Cron jobs use wall-clock time, because "every day at 3am" needs to mean 3am.
|
|
108
101
|
|
|
109
102
|
---
|
|
110
103
|
|
|
111
|
-
##
|
|
104
|
+
## How it works
|
|
112
105
|
|
|
113
106
|
```
|
|
114
|
-
schedule.yml
|
|
115
|
-
│
|
|
116
|
-
▼
|
|
117
|
-
build_heap ← parse config, validate, assign workers
|
|
118
|
-
│
|
|
119
|
-
▼
|
|
120
|
-
MinHeap<Entry> ← O(log N) push/pop, sorted by next_run_at
|
|
121
|
-
│
|
|
122
|
-
▼
|
|
123
|
-
1 scheduler loop ← single Async task, sleeps until next entry
|
|
124
|
-
│
|
|
125
|
-
▼
|
|
126
|
-
Semaphore ← limits concurrent job execution
|
|
127
|
-
│
|
|
128
|
-
▼
|
|
129
|
-
run_job ← timeout, logging, error handling
|
|
107
|
+
schedule.yml ─► build_heap ─► MinHeap<Entry> ─► scheduler loop ─► Semaphore ─► run_job
|
|
130
108
|
```
|
|
131
109
|
|
|
132
|
-
|
|
110
|
+
A single Async task sleeps until the next entry is due, then dispatches it under a semaphore that caps concurrency. Overlapping ticks are skipped and rescheduled.
|
|
111
|
+
|
|
112
|
+
The dynamic queue runs alongside it:
|
|
133
113
|
|
|
134
114
|
```
|
|
135
|
-
Producer (web/console)
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
(INSERT job) (jobs table) (IO.pipe wakeup)
|
|
115
|
+
Producer (web/console) Consumer (background worker)
|
|
116
|
+
│ │
|
|
117
|
+
▼ ▼
|
|
118
|
+
Queue::Client Queue::Store#fetch
|
|
119
|
+
push / push_in / push_at (run_at <= now)
|
|
120
|
+
│ ▲
|
|
121
|
+
▼ │
|
|
122
|
+
Queue::Store ──── SQLite (jobs) ──── SocketWaker
|
|
123
|
+
│ ▲
|
|
124
|
+
└───────► SocketNotifier ───────────────┘
|
|
125
|
+
(UNIX socket wake-up, ~80µs)
|
|
147
126
|
```
|
|
148
127
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
| Key | Required | Description |
|
|
152
|
-
|---|---|---|
|
|
153
|
-
| `class` | yes | Must include `Async::Background::Job` |
|
|
154
|
-
| `every` | one of | Interval in seconds between runs |
|
|
155
|
-
| `cron` | one of | Cron expression (parsed by Fugit) |
|
|
156
|
-
| `timeout` | no | Max execution time in seconds (default: 30) |
|
|
157
|
-
| `worker` | no | Pin to specific worker index. If omitted — `crc32(name) % total_workers` |
|
|
158
|
-
|
|
159
|
-
## SQLite Pragmas
|
|
160
|
-
|
|
161
|
-
| Pragma | Value | Why |
|
|
162
|
-
|---|---|---|
|
|
163
|
-
| `journal_mode` | WAL | Concurrent reads during writes |
|
|
164
|
-
| `synchronous` | NORMAL | Safe with WAL, lower fsync overhead |
|
|
165
|
-
| `mmap_size` | 256 MB | Fast reads ([requires proper filesystem](#docker-sqlite-requires-a-named-volume)) |
|
|
166
|
-
| `cache_size` | 16000 pages | ~64 MB page cache |
|
|
167
|
-
| `busy_timeout` | 5000 ms | Wait instead of failing on lock contention |
|
|
128
|
+
Jobs are persisted in SQLite, so a missed wake-up is never a lost job — workers also poll every 5 seconds as a safety net.
|
|
168
129
|
|
|
169
130
|
## Metrics
|
|
170
131
|
|
|
171
|
-
|
|
132
|
+
With `async-utilization` installed, per-worker stats land in shared memory at `/tmp/async-background.shm` with lock-free updates.
|
|
172
133
|
|
|
173
134
|
```ruby
|
|
174
135
|
runner.metrics.values
|
|
175
136
|
# => { total_runs: 142, total_successes: 140, total_failures: 2,
|
|
176
|
-
# total_timeouts: 0, total_skips: 5, active_jobs: 1,
|
|
177
|
-
# last_run_at: 1774445243, last_duration_ms: 1250 }
|
|
137
|
+
# total_timeouts: 0, total_skips: 5, active_jobs: 1, ... }
|
|
178
138
|
|
|
179
|
-
# Read all workers at once (no server needed)
|
|
180
139
|
Async::Background::Metrics.read_all(total_workers: 2)
|
|
181
140
|
```
|
|
182
141
|
|
|
183
|
-
Without the gem
|
|
184
|
-
|
|
185
|
-
## Why Ruby 3.3?
|
|
186
|
-
|
|
187
|
-
- Ruby 3.0 introduced Fiber Scheduler but had critical bugs
|
|
188
|
-
- Ruby 3.2 is the first production-ready release (per Samuel Williams)
|
|
189
|
-
- `io-event >= 1.14` (pulled by latest `async`) requires Ruby `>= 3.3`
|
|
142
|
+
Without the gem, metrics are silently disabled — zero overhead.
|
|
190
143
|
|
|
191
144
|
## License
|
|
192
145
|
|
data/lib/async/background/job.rb
CHANGED
|
@@ -3,6 +3,12 @@
|
|
|
3
3
|
module Async
|
|
4
4
|
module Background
|
|
5
5
|
module Job
|
|
6
|
+
DEFAULT_TIMEOUT = 120
|
|
7
|
+
|
|
8
|
+
Options = Data.define(:timeout) do
|
|
9
|
+
def initialize(timeout: DEFAULT_TIMEOUT) = super(timeout: Integer(timeout))
|
|
10
|
+
end
|
|
11
|
+
|
|
6
12
|
def self.included(base)
|
|
7
13
|
base.extend(ClassMethods)
|
|
8
14
|
end
|
|
@@ -12,17 +18,23 @@ module Async
|
|
|
12
18
|
new.perform(*args)
|
|
13
19
|
end
|
|
14
20
|
|
|
15
|
-
def perform_async(*args)
|
|
16
|
-
Async::Background::Queue.enqueue(self, *args)
|
|
21
|
+
def perform_async(*args, options: {})
|
|
22
|
+
Async::Background::Queue.enqueue(self, *args, options: options)
|
|
17
23
|
end
|
|
18
24
|
|
|
19
|
-
def perform_in(delay, *args)
|
|
20
|
-
Async::Background::Queue.enqueue_in(delay, self, *args)
|
|
25
|
+
def perform_in(delay, *args, options: {})
|
|
26
|
+
Async::Background::Queue.enqueue_in(delay, self, *args, options: options)
|
|
21
27
|
end
|
|
22
28
|
|
|
23
|
-
def perform_at(time, *args)
|
|
24
|
-
Async::Background::Queue.enqueue_at(time, self, *args)
|
|
29
|
+
def perform_at(time, *args, options: {})
|
|
30
|
+
Async::Background::Queue.enqueue_at(time, self, *args, options: options)
|
|
25
31
|
end
|
|
32
|
+
|
|
33
|
+
def options(**values)
|
|
34
|
+
@options = Options.new(**values).to_h
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def resolve_options = @options || {}
|
|
26
38
|
end
|
|
27
39
|
|
|
28
40
|
def perform(*args)
|
|
@@ -5,6 +5,8 @@ require_relative '../clock'
|
|
|
5
5
|
module Async
|
|
6
6
|
module Background
|
|
7
7
|
module Queue
|
|
8
|
+
EMPTY_OPTIONS = {}.freeze
|
|
9
|
+
|
|
8
10
|
class Client
|
|
9
11
|
include Clock
|
|
10
12
|
|
|
@@ -13,43 +15,53 @@ module Async
|
|
|
13
15
|
@notifier = notifier
|
|
14
16
|
end
|
|
15
17
|
|
|
16
|
-
def push(class_name, args = [], run_at = nil)
|
|
17
|
-
id = @store.enqueue(class_name, args, run_at)
|
|
18
|
+
def push(class_name, args = [], run_at = nil, options: {})
|
|
19
|
+
id = @store.enqueue(class_name, args, run_at, options: options)
|
|
18
20
|
@notifier&.notify_all
|
|
19
21
|
id
|
|
20
22
|
end
|
|
21
23
|
|
|
22
|
-
def push_in(delay, class_name, args = [])
|
|
24
|
+
def push_in(delay, class_name, args = [], options: {})
|
|
23
25
|
run_at = realtime_now + delay.to_f
|
|
24
|
-
push(class_name, args, run_at)
|
|
26
|
+
push(class_name, args, run_at, options: options)
|
|
25
27
|
end
|
|
26
28
|
|
|
27
|
-
def push_at(time, class_name, args = [])
|
|
29
|
+
def push_at(time, class_name, args = [], options: {})
|
|
28
30
|
run_at = time.respond_to?(:to_f) ? time.to_f : time
|
|
29
|
-
push(class_name, args, run_at)
|
|
31
|
+
push(class_name, args, run_at, options: options)
|
|
30
32
|
end
|
|
31
33
|
end
|
|
32
34
|
|
|
33
35
|
class << self
|
|
34
36
|
attr_accessor :default_client
|
|
35
37
|
|
|
36
|
-
def enqueue(job_class, *args)
|
|
38
|
+
def enqueue(job_class, *args, options: {})
|
|
37
39
|
ensure_configured!
|
|
38
|
-
|
|
40
|
+
merged = build_options(job_class, options)
|
|
41
|
+
default_client.push(resolve_class_name(job_class), args, nil, options: merged)
|
|
39
42
|
end
|
|
40
43
|
|
|
41
|
-
def enqueue_in(delay, job_class, *args)
|
|
44
|
+
def enqueue_in(delay, job_class, *args, options: {})
|
|
42
45
|
ensure_configured!
|
|
43
|
-
|
|
46
|
+
merged = build_options(job_class, options)
|
|
47
|
+
default_client.push_in(delay, resolve_class_name(job_class), args, options: merged)
|
|
44
48
|
end
|
|
45
49
|
|
|
46
|
-
def enqueue_at(time, job_class, *args)
|
|
50
|
+
def enqueue_at(time, job_class, *args, options: {})
|
|
47
51
|
ensure_configured!
|
|
48
|
-
|
|
52
|
+
merged = build_options(job_class, options)
|
|
53
|
+
default_client.push_at(time, resolve_class_name(job_class), args, options: merged)
|
|
49
54
|
end
|
|
50
55
|
|
|
51
56
|
private
|
|
52
57
|
|
|
58
|
+
def build_options(job_class, call_site_options)
|
|
59
|
+
merged = resolve_options(job_class).merge!(call_site_options.compact)
|
|
60
|
+
return EMPTY_OPTIONS if merged.empty?
|
|
61
|
+
|
|
62
|
+
Job::Options.new(**merged).to_h
|
|
63
|
+
end
|
|
64
|
+
|
|
53
65
|
def ensure_configured!
|
|
54
66
|
raise "Async::Background::Queue not configured" unless default_client
|
|
55
67
|
end
|
|
@@ -60,6 +72,12 @@ module Async
|
|
|
60
72
|
|
|
61
73
|
raise ArgumentError, "#{job_class} must include Async::Background::Job"
|
|
62
74
|
end
|
|
75
|
+
|
|
76
|
+
def resolve_options(job_class)
|
|
77
|
+
return {} unless job_class.respond_to?(:resolve_options)
|
|
78
|
+
|
|
79
|
+
job_class.resolve_options.dup
|
|
80
|
+
end
|
|
63
81
|
end
|
|
64
82
|
end
|
|
65
83
|
end
|
|
@@ -6,13 +6,11 @@ module Async
|
|
|
6
6
|
module Background
|
|
7
7
|
module Queue
|
|
8
8
|
class SocketNotifier
|
|
9
|
-
# Errors that indicate a worker is unavailable - silently skip
|
|
9
|
+
# Errors that indicate a worker is unavailable - silently skip and try the next.
|
|
10
10
|
UNAVAILABLE = [
|
|
11
11
|
Errno::ENOENT, # Socket file doesn't exist (worker hasn't started yet)
|
|
12
12
|
Errno::ECONNREFUSED, # File exists but no one listening (worker died)
|
|
13
13
|
Errno::EPIPE, # Connection broken during write
|
|
14
|
-
Errno::EAGAIN, # Socket buffer full - wake-up already queued
|
|
15
|
-
IO::WaitWritable, # Same as EAGAIN on some platforms
|
|
16
14
|
Errno::ECONNRESET # Connection reset by peer
|
|
17
15
|
].freeze
|
|
18
16
|
|
|
@@ -22,8 +20,12 @@ module Async
|
|
|
22
20
|
end
|
|
23
21
|
|
|
24
22
|
def notify_all
|
|
25
|
-
|
|
26
|
-
|
|
23
|
+
return if @total_workers <= 0
|
|
24
|
+
|
|
25
|
+
start = rand(@total_workers)
|
|
26
|
+
@total_workers.times do |i|
|
|
27
|
+
worker_index = ((start + i) % @total_workers) + 1
|
|
28
|
+
return if notify_one(worker_index)
|
|
27
29
|
end
|
|
28
30
|
end
|
|
29
31
|
|
|
@@ -37,14 +39,12 @@ module Async
|
|
|
37
39
|
ensure
|
|
38
40
|
sock.close rescue nil
|
|
39
41
|
end
|
|
42
|
+
true
|
|
40
43
|
rescue *UNAVAILABLE
|
|
41
|
-
|
|
42
|
-
# The job is already in the database. The worker will:
|
|
43
|
-
# - Pick it up on next poll (within QUEUE_POLL_INTERVAL seconds), or
|
|
44
|
-
# - Pick it up when it starts/restarts via normal fetch loop
|
|
44
|
+
false
|
|
45
45
|
rescue => e
|
|
46
|
-
# Unexpected error - log but don't crash the enqueue operation
|
|
47
46
|
Console.logger.warn(self) { "SocketNotifier#notify_one(#{worker_index}) failed: #{e.class} #{e.message}" } rescue nil
|
|
47
|
+
false
|
|
48
48
|
end
|
|
49
49
|
|
|
50
50
|
def socket_path(worker_index)
|
|
@@ -15,13 +15,14 @@ module Async
|
|
|
15
15
|
id INTEGER PRIMARY KEY,
|
|
16
16
|
class_name TEXT NOT NULL,
|
|
17
17
|
args TEXT NOT NULL DEFAULT '[]',
|
|
18
|
+
options TEXT,
|
|
18
19
|
status TEXT NOT NULL DEFAULT 'pending',
|
|
19
20
|
created_at REAL NOT NULL,
|
|
20
21
|
run_at REAL NOT NULL,
|
|
21
22
|
locked_by INTEGER,
|
|
22
23
|
locked_at REAL
|
|
23
24
|
);
|
|
24
|
-
CREATE INDEX IF NOT EXISTS
|
|
25
|
+
CREATE INDEX IF NOT EXISTS idx_jobs_pending ON jobs(run_at, id) WHERE status = 'pending';
|
|
25
26
|
SQL
|
|
26
27
|
|
|
27
28
|
MMAP_SIZE = 268_435_456
|
|
@@ -45,6 +46,7 @@ module Async
|
|
|
45
46
|
def initialize(path: self.class.default_path, mmap: true)
|
|
46
47
|
@path = path
|
|
47
48
|
@mmap = mmap
|
|
49
|
+
@pragma_sql = PRAGMAS.call(mmap ? MMAP_SIZE : 0).freeze
|
|
48
50
|
@db = nil
|
|
49
51
|
@schema_checked = false
|
|
50
52
|
@last_cleanup_at = nil
|
|
@@ -54,17 +56,18 @@ module Async
|
|
|
54
56
|
require_sqlite3
|
|
55
57
|
db = SQLite3::Database.new(@path)
|
|
56
58
|
db.execute('PRAGMA busy_timeout = 5000')
|
|
57
|
-
db.execute_batch(
|
|
59
|
+
db.execute_batch(@pragma_sql)
|
|
58
60
|
db.execute_batch(SCHEMA)
|
|
59
61
|
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
|
60
62
|
db.close
|
|
61
63
|
@schema_checked = true
|
|
62
64
|
end
|
|
63
65
|
|
|
64
|
-
def enqueue(class_name, args = [], run_at = nil)
|
|
66
|
+
def enqueue(class_name, args = [], run_at = nil, options: {})
|
|
65
67
|
ensure_connection
|
|
66
68
|
run_at ||= realtime_now
|
|
67
|
-
|
|
69
|
+
options_json = options.empty? ? nil : JSON.generate(options)
|
|
70
|
+
@enqueue_stmt.execute(class_name, JSON.generate(args), options_json, realtime_now, run_at)
|
|
68
71
|
@db.last_insert_row_id
|
|
69
72
|
end
|
|
70
73
|
|
|
@@ -72,14 +75,20 @@ module Async
|
|
|
72
75
|
ensure_connection
|
|
73
76
|
now = realtime_now
|
|
74
77
|
@db.execute("BEGIN IMMEDIATE")
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
+
|
|
79
|
+
begin
|
|
80
|
+
results = @fetch_stmt.execute(worker_id, now, now)
|
|
81
|
+
row = results.first
|
|
82
|
+
ensure
|
|
83
|
+
@fetch_stmt.reset! rescue nil
|
|
84
|
+
end
|
|
85
|
+
|
|
78
86
|
@db.execute("COMMIT")
|
|
79
87
|
return unless row
|
|
80
88
|
|
|
81
89
|
maybe_cleanup
|
|
82
|
-
|
|
90
|
+
options = row[3] ? JSON.parse(row[3], symbolize_names: true) : {}
|
|
91
|
+
{ id: row[0], class_name: row[1], args: JSON.parse(row[2]), options: options }
|
|
83
92
|
rescue
|
|
84
93
|
@db.execute("ROLLBACK") rescue nil
|
|
85
94
|
raise
|
|
@@ -130,10 +139,11 @@ module Async
|
|
|
130
139
|
finalize_statements
|
|
131
140
|
@db = SQLite3::Database.new(@path)
|
|
132
141
|
@db.execute('PRAGMA busy_timeout = 5000')
|
|
133
|
-
@db.execute_batch(
|
|
142
|
+
@db.execute_batch(@pragma_sql)
|
|
134
143
|
|
|
135
144
|
unless @schema_checked
|
|
136
145
|
@db.execute_batch(SCHEMA)
|
|
146
|
+
@db.execute("ALTER TABLE jobs ADD COLUMN options TEXT") rescue nil
|
|
137
147
|
@schema_checked = true
|
|
138
148
|
end
|
|
139
149
|
|
|
@@ -143,7 +153,7 @@ module Async
|
|
|
143
153
|
|
|
144
154
|
def prepare_statements
|
|
145
155
|
@enqueue_stmt = @db.prepare(
|
|
146
|
-
"INSERT INTO jobs (class_name, args, created_at, run_at) VALUES (?, ?, ?, ?)"
|
|
156
|
+
"INSERT INTO jobs (class_name, args, options, created_at, run_at) VALUES (?, ?, ?, ?, ?)"
|
|
147
157
|
)
|
|
148
158
|
|
|
149
159
|
@fetch_stmt = @db.prepare(<<~SQL)
|
|
@@ -155,7 +165,7 @@ module Async
|
|
|
155
165
|
ORDER BY run_at, id
|
|
156
166
|
LIMIT 1
|
|
157
167
|
)
|
|
158
|
-
RETURNING id, class_name, args
|
|
168
|
+
RETURNING id, class_name, args, options
|
|
159
169
|
SQL
|
|
160
170
|
|
|
161
171
|
@complete_stmt = @db.prepare("UPDATE jobs SET status = 'done' WHERE id = ?")
|
|
@@ -170,11 +180,11 @@ module Async
|
|
|
170
180
|
end
|
|
171
181
|
|
|
172
182
|
def finalize_statements
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
stmt&.close rescue nil
|
|
176
|
-
instance_variable_set(:"@#{name}", nil)
|
|
183
|
+
[@enqueue_stmt, @fetch_stmt, @complete_stmt, @fail_stmt, @requeue_stmt, @cleanup_stmt].each do |s|
|
|
184
|
+
s&.close rescue next
|
|
177
185
|
end
|
|
186
|
+
|
|
187
|
+
@enqueue_stmt = @fetch_stmt = @complete_stmt = @fail_stmt = @requeue_stmt = @cleanup_stmt = nil
|
|
178
188
|
end
|
|
179
189
|
|
|
180
190
|
def maybe_cleanup
|
|
@@ -188,7 +198,6 @@ module Async
|
|
|
188
198
|
@db.execute("PRAGMA incremental_vacuum")
|
|
189
199
|
end
|
|
190
200
|
end
|
|
191
|
-
|
|
192
201
|
end
|
|
193
202
|
end
|
|
194
203
|
end
|
|
@@ -114,11 +114,13 @@ module Async
|
|
|
114
114
|
class_name = job[:class_name]
|
|
115
115
|
args = job[:args]
|
|
116
116
|
klass = resolve_job_class(class_name)
|
|
117
|
+
opts = Job::Options.new(**job[:options])
|
|
118
|
+
timeout = opts.timeout
|
|
117
119
|
|
|
118
120
|
metrics.job_started(nil)
|
|
119
121
|
t = monotonic_now
|
|
120
122
|
|
|
121
|
-
job_task.with_timeout(
|
|
123
|
+
job_task.with_timeout(timeout) { klass.perform_now(*args) }
|
|
122
124
|
|
|
123
125
|
duration = monotonic_now - t
|
|
124
126
|
metrics.job_finished(nil, duration)
|
|
@@ -130,7 +132,7 @@ module Async
|
|
|
130
132
|
rescue ::Async::TimeoutError
|
|
131
133
|
metrics.job_timed_out(nil)
|
|
132
134
|
@queue_store.fail(job[:id])
|
|
133
|
-
logger.error('Async::Background') { "queue(#{class_name}): timed out" }
|
|
135
|
+
logger.error('Async::Background') { "queue(#{class_name}): timed out after #{timeout}s" }
|
|
134
136
|
rescue => e
|
|
135
137
|
metrics.job_failed(nil, e)
|
|
136
138
|
@queue_store.fail(job[:id])
|
|
@@ -170,13 +172,16 @@ module Async
|
|
|
170
172
|
if entry.running
|
|
171
173
|
logger.warn('Async::Background') { "#{entry.name}: skipped, previous run still active" }
|
|
172
174
|
metrics.job_skipped(entry)
|
|
173
|
-
|
|
174
|
-
entry
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
175
|
+
entry.reschedule(monotonic_now)
|
|
176
|
+
heap.replace_top(entry)
|
|
177
|
+
next
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
entry.running = true
|
|
181
|
+
semaphore.async do |job_task|
|
|
182
|
+
run_job(job_task, entry)
|
|
183
|
+
ensure
|
|
184
|
+
entry.running = false
|
|
180
185
|
end
|
|
181
186
|
|
|
182
187
|
entry.reschedule(monotonic_now)
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: async-background
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.6.
|
|
4
|
+
version: 0.6.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Roman Hajdarov
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-04-
|
|
11
|
+
date: 2026-04-16 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: async
|