async-background 0.5.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +160 -0
- data/LICENSE +21 -0
- data/README.md +193 -0
- data/async-background.gemspec +43 -0
- data/lib/async/background/queue/notifier.rb +18 -6
- data/lib/async/background/queue/store.rb +2 -0
- data/lib/async/background/version.rb +1 -1
- data/lib/async/background.rb +1 -0
- metadata +6 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 26e459d9e86a150ff8b3126ce8899ac9beb60355e3313d5510c1c0c6dec0bfd7
|
|
4
|
+
data.tar.gz: 93a52a9139f902e1acee7b58ea98dceadba2219537329b785f3a103009f72401
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3dcfcc9089b2bb8b5b3fb01341039fa1dea32896f6e3845f137bd9e307b366b2716208d4ecb533b266cf31c3b0466535817d7dc9d6c1a0876e113519417a0512
|
|
7
|
+
data.tar.gz: ece53f421338c16a02548754a6c1a2e8a62988659a83f4575e8ad23d01fd987958e6d3fb94360607444ff34a4dc0fb8844fdfdbca9228c2c8af09b1aec486b36
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.5.1
|
|
4
|
+
|
|
5
|
+
### Testing Infrastructure
|
|
6
|
+
- **Comprehensive CI setup** — full Docker-based integration testing environment with `Dockerfile.ci`, `docker-compose.ci.yml`, and `Gemfile.ci`
|
|
7
|
+
- **End-to-end scenario testing** — new `ci/scenario_test.rb` validates real-world scenarios with forked workers:
|
|
8
|
+
- Normal execution of fast/slow/failing jobs across multiple workers
|
|
9
|
+
- Crash recovery after SIGKILL with automatic job pickup by remaining workers
|
|
10
|
+
- No duplicate execution guarantees under worker crashes
|
|
11
|
+
- Proper job distribution validation across worker pool
|
|
12
|
+
- **Test fixtures** — dedicated `ci/fixtures/jobs.rb` and `ci/fixtures/schedule.yml` for scenario testing
|
|
13
|
+
|
|
14
|
+
### Bug Fixes
|
|
15
|
+
- **SQLite busy timeout** — added `PRAGMA busy_timeout = 5000` to `Queue::Store` to prevent `SQLITE_BUSY` errors under concurrent multi-process database access
|
|
16
|
+
- **Enhanced Queue::Notifier error handling** — restructured IO error handling with clearer categorization:
|
|
17
|
+
- `WRITE_DROPPED` for write failures (`IO::WaitWritable`, `Errno::EAGAIN`, `IOError`, `Errno::EPIPE`) — all non-fatal as job is already in store
|
|
18
|
+
- `READ_EXHAUSTED` for read exhaustion (`IO::WaitReadable`, `EOFError`, `IOError`) — normal drain completion
|
|
19
|
+
- Added explanatory comments for each error type and handling strategy
|
|
20
|
+
|
|
21
|
+
## 0.5.0
|
|
22
|
+
|
|
23
|
+
### Features
|
|
24
|
+
- **Delayed jobs** — full support for scheduling jobs in the future
|
|
25
|
+
- `Queue::Client#push_in(delay, class_name, args)` — enqueue with delay in seconds
|
|
26
|
+
- `Queue::Client#push_at(time, class_name, args)` — enqueue at a specific time
|
|
27
|
+
- `Queue.enqueue_in(delay, job_class, *args)` — class-level delayed enqueue
|
|
28
|
+
- `Queue.enqueue_at(time, job_class, *args)` — class-level scheduled enqueue
|
|
29
|
+
- New `run_at` column in SQLite `jobs` table — jobs are only fetched when `run_at <= now`
|
|
30
|
+
- **Job module** — Sidekiq-like `include Async::Background::Job` interface
|
|
31
|
+
- `perform_async(*args)` — immediate queue execution
|
|
32
|
+
- `perform_in(delay, *args)` — delayed execution after N seconds
|
|
33
|
+
- `perform_at(time, *args)` — scheduled execution at a specific time
|
|
34
|
+
- Instance-level `#perform` with class-level `perform_now` delegation
|
|
35
|
+
- **Clock module** — shared `monotonic_now` / `realtime_now` helpers extracted into `Async::Background::Clock`, included by `Runner`, `Queue::Store`, and `Queue::Client`
|
|
36
|
+
|
|
37
|
+
### Bug Fixes
|
|
38
|
+
- **Runner: incorrect task in `with_timeout`** — `semaphore.async { |job_task| ... }` now correctly receives the child task instead of capturing the parent `task` from the outer scope. Previously, `with_timeout` was applied to the parent task, which could cancel unrelated work
|
|
39
|
+
|
|
40
|
+
### Improvements
|
|
41
|
+
- **Job API: `#perform` instead of `#perform_now`** — job classes now define `#perform` instance method. The class-level `perform_now` creates instance and calls `#perform`, aligning with ActiveJob / Sidekiq conventions
|
|
42
|
+
- Updated error messages: validation failures now suggest `must include Async::Background::Job` instead of `must implement .perform_now`
|
|
43
|
+
- `Queue::Client` — extracted private `ensure_configured!` and `resolve_class_name` methods for cleaner validation and class name resolution logic
|
|
44
|
+
- `Queue::Notifier` — extracted `IO_ERRORS` constant (`IO::WaitReadable`, `EOFError`, `IOError`) for cleaner `rescue` in `drain`
|
|
45
|
+
- `Queue::Store` — replaced index `idx_jobs_status_id(status, id)` with `idx_jobs_status_run_at_id(status, run_at, id)` for efficient delayed job lookups
|
|
46
|
+
- `Queue::Store` — `fetch` SQL now uses `WHERE status = 'pending' AND run_at <= ?` with `ORDER BY run_at, id` to process jobs in scheduled order
|
|
47
|
+
- Removed duplicated `monotonic_now` / `realtime_now` from `Runner` and `Store` — now provided by `Clock` module
|
|
48
|
+
- Updated documentation: README (Job module examples, Queue architecture diagram, Clock section), GET_STARTED (delayed jobs guide, Job module usage, minimal queue-only example)
|
|
49
|
+
|
|
50
|
+
## 0.4.5
|
|
51
|
+
|
|
52
|
+
### Breaking Changes
|
|
53
|
+
- `PRAGMAS` is now a frozen lambda `PRAGMAS.call(mmap_size)` instead of a static string — if you referenced this constant directly, update your code
|
|
54
|
+
|
|
55
|
+
### Features
|
|
56
|
+
- New `queue_mmap:` parameter on `Runner` (default: `true`) — allows disabling SQLite mmap for environments where it's unsafe (Docker overlay2)
|
|
57
|
+
- New `mmap:` parameter on `Queue::Store` (default: `true`) — controls `PRAGMA mmap_size` (256 MB when enabled, 0 when disabled)
|
|
58
|
+
- Public `attr_reader :queue_store` on `Runner` — eliminates need for `instance_variable_get` when sharing Store with Client
|
|
59
|
+
|
|
60
|
+
### Bug Fixes
|
|
61
|
+
- **CRITICAL: fetch race condition** — wrapped `UPDATE ... RETURNING` in `BEGIN IMMEDIATE` transaction to prevent two workers from picking up the same job simultaneously
|
|
62
|
+
- **CRITICAL: mmap + Docker overlay2** — `overlay2` filesystem does not guarantee `write()`/`mmap()` coherence, causing SQLite WAL corruption under concurrent multi-process access. mmap is now configurable via `queue_mmap: false` instead of being hardcoded. Documented proper Docker setup with named volumes in `docs/GET_STARTED.md`
|
|
63
|
+
- **`PRAGMA optimize` on shutdown** — wrapped in `rescue nil` to prevent `SQLite3::BusyException` when another process holds the write lock during graceful shutdown
|
|
64
|
+
- **`PRAGMA incremental_vacuum` was a no-op** — added `PRAGMA auto_vacuum = INCREMENTAL` to schema. Without it, `incremental_vacuum` does nothing. Note: only takes effect on newly created databases; existing databases require a one-time `VACUUM`
|
|
65
|
+
|
|
66
|
+
### Improvements
|
|
67
|
+
- Replaced index `idx_jobs_status(status)` with composite `idx_jobs_status_id(status, id)` — eliminates sort step in `fetch` query (`ORDER BY id LIMIT 1` is now a direct B-tree lookup)
|
|
68
|
+
- Fixed `finalize_statements` — changed `%i[@enqueue_stmt ...]` to `%i[enqueue_stmt ...]` with `:"@#{name}"` interpolation for idiomatic `instance_variable_get`/`set` usage
|
|
69
|
+
- Added documentation: `README.md` (concise, with warning markers) and `docs/GET_STARTED.md` (step-by-step guide covering schedule config, Falcon integration, Docker setup, dynamic queue)
|
|
70
|
+
|
|
71
|
+
## 0.4.0
|
|
72
|
+
|
|
73
|
+
### Features
|
|
74
|
+
- **Dynamic job queue** — enqueue jobs at runtime from any process (web, console, rake) with automatic execution by background workers
|
|
75
|
+
- `Queue::Store` — SQLite-backed persistent storage with WAL mode, prepared statements, and optimized pragmas
|
|
76
|
+
- `Queue::Notifier` — `IO.pipe`-based zero-cost wakeup between producer and consumer processes (no polling)
|
|
77
|
+
- `Queue::Client` — public API: `Async::Background::Queue.enqueue(JobClass, *args)`
|
|
78
|
+
- Automatic recovery of stale `running` jobs on worker restart
|
|
79
|
+
- Periodic cleanup of completed jobs (piggyback on fetch, every 5 minutes)
|
|
80
|
+
- `PRAGMA incremental_vacuum` when cleanup removes 100+ rows
|
|
81
|
+
- Worker isolation via `ISOLATION_FORKS` env variable — exclude specific workers from queue processing
|
|
82
|
+
- Custom database path via `queue_db_path` parameter
|
|
83
|
+
- Requires optional `sqlite3` gem (`~> 2.0`) — not included by default, must be added to Gemfile explicitly
|
|
84
|
+
- New Runner parameters: `queue_notifier:` and `queue_db_path:`
|
|
85
|
+
|
|
86
|
+
### Improvements
|
|
87
|
+
- Unified `monotonic_now` usage across `run_job` and `run_queue_job` (was using direct `Process.clock_gettime` call in `run_job`)
|
|
88
|
+
- `Queue::Notifier#drain` — moved `rescue` inside the loop to avoid stack unwinding on each drain cycle
|
|
89
|
+
|
|
90
|
+
## 0.3.0
|
|
91
|
+
|
|
92
|
+
### Features
|
|
93
|
+
- Added optional metrics collection system using shared memory
|
|
94
|
+
- New `Metrics` class with worker-specific performance tracking
|
|
95
|
+
- Public API: `runner.metrics.enabled?`, `runner.metrics.values`, `Metrics.read_all()`
|
|
96
|
+
- Tracks total runs, successes, failures, timeouts, skips, active jobs, and execution times
|
|
97
|
+
- Requires optional `async-utilization` gem dependency
|
|
98
|
+
- Metrics stored in `/tmp/async-background.shm` with lock-free updates per worker
|
|
99
|
+
|
|
100
|
+
## 0.2.6
|
|
101
|
+
|
|
102
|
+
### Improvements
|
|
103
|
+
- Micro-optimization in `wait_with_shutdown` method: use passed `task` parameter instead of `Async::Task.current` for better consistency and slight performance improvement
|
|
104
|
+
|
|
105
|
+
## 0.2.5
|
|
106
|
+
|
|
107
|
+
### Features
|
|
108
|
+
- Added graceful shutdown via signal handlers for SIGINT and SIGTERM
|
|
109
|
+
- Enhanced process lifecycle management with proper signal handling using `Signal.trap` and IO.pipe for async communication
|
|
110
|
+
- Improved robustness for production deployments and container orchestration
|
|
111
|
+
- Updated dependencies to work with latest Async 2.x API (removed deprecated `:parent` parameter usage)
|
|
112
|
+
|
|
113
|
+
## 0.2.4
|
|
114
|
+
|
|
115
|
+
### Improvements
|
|
116
|
+
- Removed hardcoded version warning from main module (was checking against fixed list: 0.1.0, 0.2.2, 0.2.3). Use semantic versioning with pre-release suffixes for unstable versions (e.g., 0.3.0.alpha1) instead
|
|
117
|
+
- Removed hardcoded stable versions list from gemspec description — metadata should describe functionality, not versioning
|
|
118
|
+
- Changed `while true` to idiomatic `loop do` in run method
|
|
119
|
+
- Added `Gemfile.lock` to .gitignore (gems should not commit lockfile)
|
|
120
|
+
- Updated README: clarified that job class must respond to `.perform_now` class method (removed confusing mention of instance `#perform`)
|
|
121
|
+
|
|
122
|
+
## 0.2.2
|
|
123
|
+
|
|
124
|
+
### Bug Fixes
|
|
125
|
+
- **CRITICAL**: Removed logger parameter from Runner initialize (was unused). Fixed initialization to use Console.logger directly which now properly initializes in forked processes with correct context
|
|
126
|
+
|
|
127
|
+
## 0.2.1
|
|
128
|
+
|
|
129
|
+
### Bug Fixes
|
|
130
|
+
- **CRITICAL**: Added missing `require 'console'` in main module. Logger was nil because Console gem was not imported, causing `undefined method 'info' for nil` errors on worker initialization
|
|
131
|
+
|
|
132
|
+
## 0.2.0
|
|
133
|
+
|
|
134
|
+
### Bug Fixes
|
|
135
|
+
- **CRITICAL**: Removed hidden ActiveSupport dependency. Replaced `safe_constantize` with `Object.const_get` + `NameError` handling
|
|
136
|
+
- **CRITICAL**: Fixed validator mismatch: now validates `.perform_now` (class method) instead of `.perform` (instance method)
|
|
137
|
+
- **CRITICAL**: Fixed race condition where entry could disappear from heap during execution. `reschedule` and `heap.push` now always execute after job processing
|
|
138
|
+
- Added full exception backtrace to error logs for production debugging
|
|
139
|
+
- Improved YAML security by removing `Symbol` from `permitted_classes`
|
|
140
|
+
- Removed Mutex from graceful shutdown (anti-pattern in Async). Boolean assignment is atomic in MRI
|
|
141
|
+
|
|
142
|
+
### Features
|
|
143
|
+
- Added optional `logger` parameter to Runner constructor for custom loggers (Rails.logger, etc.)
|
|
144
|
+
- Added `stop()` method for graceful shutdown
|
|
145
|
+
- Added `running?()` method to check scheduler status
|
|
146
|
+
|
|
147
|
+
### Breaking Changes
|
|
148
|
+
- Job class validation now checks for `.perform_now` class method (was checking for `.perform` instance method)
|
|
149
|
+
|
|
150
|
+
## 0.1.0
|
|
151
|
+
|
|
152
|
+
- Initial release
|
|
153
|
+
- Single event loop with min-heap timer (O(log N) scheduling)
|
|
154
|
+
- Skip overlapping execution
|
|
155
|
+
- Startup jitter to prevent thundering herd
|
|
156
|
+
- Monotonic clock for interval jobs, wall clock for cron jobs
|
|
157
|
+
- Deterministic worker sharding via Zlib.crc32
|
|
158
|
+
- Semaphore-based concurrency control
|
|
159
|
+
- Per-job timeout protection
|
|
160
|
+
- Structured logging via Console
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Haydarov Roman
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# Async::Background
|
|
2
|
+
|
|
3
|
+
A lightweight, production-grade cron/interval scheduler for Ruby's [Async](https://github.com/socketry/async) ecosystem. Designed for [Falcon](https://github.com/socketry/falcon) but works with any Async-based application.
|
|
4
|
+
|
|
5
|
+
## What It Does
|
|
6
|
+
|
|
7
|
+
- **Cron & interval scheduling** — single event loop + min-heap, scales to hundreds of jobs
|
|
8
|
+
- **Dynamic job queue** — enqueue jobs at runtime via SQLite, pick up by background workers
|
|
9
|
+
- **Delayed jobs** — schedule jobs for future execution with `perform_in` / `perform_at` (Sidekiq-like API)
|
|
10
|
+
- **Multi-process safe** — deterministic worker sharding via `Zlib.crc32`, no duplicate execution
|
|
11
|
+
- **Skip overlapping** — if a job is still running when its next tick arrives, the tick is skipped
|
|
12
|
+
- **Timeout protection** — per-job configurable timeout via `Async::Task#with_timeout`
|
|
13
|
+
- **Startup jitter** — random delay to prevent thundering herd after restart
|
|
14
|
+
- **Optional metrics** — shared memory performance tracking with `async-utilization`
|
|
15
|
+
|
|
16
|
+
## Requirements
|
|
17
|
+
|
|
18
|
+
- **Ruby >= 3.3** — Fiber Scheduler production-ready ([why?](#why-ruby-33))
|
|
19
|
+
- **Async ~> 2.0** — Fiber Scheduler-based concurrency
|
|
20
|
+
- **Fugit ~> 1.0** — cron expression parsing
|
|
21
|
+
|
|
22
|
+
## Installation
|
|
23
|
+
|
|
24
|
+
```ruby
|
|
25
|
+
# Gemfile
|
|
26
|
+
gem "async-background"
|
|
27
|
+
|
|
28
|
+
# Optional
|
|
29
|
+
gem "sqlite3", "~> 2.0" # for dynamic job queue
|
|
30
|
+
gem "async-utilization", "~> 0.3" # for metrics
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## ➡️ [Get Started](docs/GET_STARTED.md)
|
|
34
|
+
|
|
35
|
+
Step-by-step setup guide: schedule config, Falcon integration, Docker, dynamic queue, delayed jobs.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Quick Example: Job Module
|
|
40
|
+
|
|
41
|
+
Include `Async::Background::Job` for a Sidekiq-like interface:
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
class SendEmailJob
|
|
45
|
+
include Async::Background::Job
|
|
46
|
+
|
|
47
|
+
def perform(user_id, template)
|
|
48
|
+
user = User.find(user_id)
|
|
49
|
+
Mailer.send(user, template)
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Immediate execution in the queue
|
|
54
|
+
SendEmailJob.perform_async(user_id, "welcome")
|
|
55
|
+
|
|
56
|
+
# Execute after 5 minutes
|
|
57
|
+
SendEmailJob.perform_in(300, user_id, "reminder")
|
|
58
|
+
|
|
59
|
+
# Execute at a specific time
|
|
60
|
+
SendEmailJob.perform_at(Time.new(2026, 4, 1, 9, 0, 0), user_id, "scheduled")
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Or use the lower-level API directly:
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
Async::Background::Queue.enqueue(SendEmailJob, user_id, "welcome")
|
|
67
|
+
Async::Background::Queue.enqueue_in(300, SendEmailJob, user_id, "reminder")
|
|
68
|
+
Async::Background::Queue.enqueue_at(Time.new(2026, 4, 1, 9, 0, 0), SendEmailJob, user_id, "scheduled")
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## ⚠️ Important Notes
|
|
74
|
+
|
|
75
|
+
### Docker: SQLite requires a named volume
|
|
76
|
+
|
|
77
|
+
The SQLite database **must not** live on Docker's `overlay2` filesystem. The `overlay2` driver breaks coherence between `write()` and `mmap()`, which corrupts SQLite WAL under concurrent access.
|
|
78
|
+
|
|
79
|
+
```yaml
|
|
80
|
+
# docker-compose.yml
|
|
81
|
+
volumes:
|
|
82
|
+
- queue-data:/app/tmp/queue # ← named volume, NOT overlay2
|
|
83
|
+
|
|
84
|
+
volumes:
|
|
85
|
+
queue-data:
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Without this, you will get database crashes in multi-process mode. See [Get Started → Step 3](docs/GET_STARTED.md#step-3-docker) for details.
|
|
89
|
+
|
|
90
|
+
### Fork safety
|
|
91
|
+
|
|
92
|
+
SQLite connections **must not** cross `fork()` boundaries. Always open connections **after** fork (inside `container.run` block), never before. The gem handles this internally via lazy `ensure_connection`, but if you create a `Queue::Store` manually for schema setup, close it before fork:
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
store = Async::Background::Queue::Store.new(path: db_path)
|
|
96
|
+
store.ensure_database!
|
|
97
|
+
store.close # ← before fork
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
### Clock handling
|
|
101
|
+
|
|
102
|
+
The `Clock` module provides shared time helpers used across the codebase:
|
|
103
|
+
|
|
104
|
+
- **`monotonic_now`** (`CLOCK_MONOTONIC`) — for in-process intervals and durations, immune to NTP drift / wall-clock jumps
|
|
105
|
+
- **`realtime_now`** (`CLOCK_REALTIME`) — for persisted timestamps (SQLite `run_at`, `created_at`, `locked_at`)
|
|
106
|
+
|
|
107
|
+
Interval jobs use monotonic clock. Cron jobs use `Time.now` because "every day at 3am" must respect real time. These are different clocks by design.
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## Architecture
|
|
112
|
+
|
|
113
|
+
```
|
|
114
|
+
schedule.yml
|
|
115
|
+
│
|
|
116
|
+
▼
|
|
117
|
+
build_heap ← parse config, validate, assign workers
|
|
118
|
+
│
|
|
119
|
+
▼
|
|
120
|
+
MinHeap<Entry> ← O(log N) push/pop, sorted by next_run_at
|
|
121
|
+
│
|
|
122
|
+
▼
|
|
123
|
+
1 scheduler loop ← single Async task, sleeps until next entry
|
|
124
|
+
│
|
|
125
|
+
▼
|
|
126
|
+
Semaphore ← limits concurrent job execution
|
|
127
|
+
│
|
|
128
|
+
▼
|
|
129
|
+
run_job ← timeout, logging, error handling
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
### Queue Architecture
|
|
133
|
+
|
|
134
|
+
```
|
|
135
|
+
Producer (web/console) Consumer (background worker)
|
|
136
|
+
│ │
|
|
137
|
+
▼ ▼
|
|
138
|
+
Queue::Client Queue::Store.fetch
|
|
139
|
+
│ (WHERE run_at <= now)
|
|
140
|
+
├─ push(class, args, run_at) │
|
|
141
|
+
├─ push_in(delay, class, args) ▼
|
|
142
|
+
└─ push_at(time, class, args) run_queue_job
|
|
143
|
+
│ │
|
|
144
|
+
▼ ▼
|
|
145
|
+
Queue::Store ──── SQLite ──── Queue::Notifier
|
|
146
|
+
(INSERT job) (jobs table) (IO.pipe wakeup)
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
## Schedule Config
|
|
150
|
+
|
|
151
|
+
| Key | Required | Description |
|
|
152
|
+
|---|---|---|
|
|
153
|
+
| `class` | yes | Must include `Async::Background::Job` |
|
|
154
|
+
| `every` | one of | Interval in seconds between runs |
|
|
155
|
+
| `cron` | one of | Cron expression (parsed by Fugit) |
|
|
156
|
+
| `timeout` | no | Max execution time in seconds (default: 30) |
|
|
157
|
+
| `worker` | no | Pin to specific worker index. If omitted — `crc32(name) % total_workers` |
|
|
158
|
+
|
|
159
|
+
## SQLite Pragmas
|
|
160
|
+
|
|
161
|
+
| Pragma | Value | Why |
|
|
162
|
+
|---|---|---|
|
|
163
|
+
| `journal_mode` | WAL | Concurrent reads during writes |
|
|
164
|
+
| `synchronous` | NORMAL | Safe with WAL, lower fsync overhead |
|
|
165
|
+
| `mmap_size` | 256 MB | Fast reads ([requires proper filesystem](#docker-sqlite-requires-a-named-volume)) |
|
|
166
|
+
| `cache_size` | 16000 pages | ~64 MB page cache |
|
|
167
|
+
| `busy_timeout` | 5000 ms | Wait instead of failing on lock contention |
|
|
168
|
+
|
|
169
|
+
## Metrics
|
|
170
|
+
|
|
171
|
+
When `async-utilization` gem is available, metrics are collected in shared memory (`/tmp/async-background.shm`) with lock-free updates per worker.
|
|
172
|
+
|
|
173
|
+
```ruby
|
|
174
|
+
runner.metrics.values
|
|
175
|
+
# => { total_runs: 142, total_successes: 140, total_failures: 2,
|
|
176
|
+
# total_timeouts: 0, total_skips: 5, active_jobs: 1,
|
|
177
|
+
# last_run_at: 1774445243, last_duration_ms: 1250 }
|
|
178
|
+
|
|
179
|
+
# Read all workers at once (no server needed)
|
|
180
|
+
Async::Background::Metrics.read_all(total_workers: 2)
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Without the gem — metrics are silently disabled, zero overhead.
|
|
184
|
+
|
|
185
|
+
## Why Ruby 3.3?
|
|
186
|
+
|
|
187
|
+
- Ruby 3.0 introduced Fiber Scheduler but had critical bugs
|
|
188
|
+
- Ruby 3.2 is the first production-ready release (per Samuel Williams)
|
|
189
|
+
- `io-event >= 1.14` (pulled by latest `async`) requires Ruby `>= 3.3`
|
|
190
|
+
|
|
191
|
+
## License
|
|
192
|
+
|
|
193
|
+
MIT
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'lib/async/background/version'
|
|
4
|
+
|
|
5
|
+
Gem::Specification.new do |spec|
|
|
6
|
+
spec.name = 'async-background'
|
|
7
|
+
spec.version = Async::Background::VERSION
|
|
8
|
+
spec.authors = ['Roman Hajdarov']
|
|
9
|
+
spec.email = ['romnhajdarov@gmail.com']
|
|
10
|
+
|
|
11
|
+
spec.summary = 'Lightweight heap-based cron/interval scheduler for Async.'
|
|
12
|
+
spec.description = 'A production-grade lightweight scheduler built on top of Async. ' \
|
|
13
|
+
'Single event loop with min-heap timer, skip-overlapping execution, ' \
|
|
14
|
+
'jitter, monotonic clock intervals, semaphore concurrency control, ' \
|
|
15
|
+
'and deterministic worker sharding. Designed for Falcon but works ' \
|
|
16
|
+
'with any Async-based application.'
|
|
17
|
+
|
|
18
|
+
spec.homepage = 'https://github.com/roman-haidarov/async-background'
|
|
19
|
+
spec.license = 'MIT'
|
|
20
|
+
|
|
21
|
+
spec.metadata = {
|
|
22
|
+
'source_code_uri' => 'https://github.com/roman-haidarov/async-background',
|
|
23
|
+
'changelog_uri' => 'https://github.com/roman-haidarov/async-background/blob/main/CHANGELOG.md',
|
|
24
|
+
'bug_tracker_uri' => 'https://github.com/roman-haidarov/async-background/issues'
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
spec.files = Dir.glob('lib/**/*', base: __dir__) +
|
|
28
|
+
%w[README.md CHANGELOG.md LICENSE async-background.gemspec]
|
|
29
|
+
spec.require_paths = ['lib']
|
|
30
|
+
|
|
31
|
+
spec.required_ruby_version = '>= 3.3'
|
|
32
|
+
|
|
33
|
+
spec.add_dependency 'async', '~> 2.0'
|
|
34
|
+
spec.add_dependency 'console', '~> 1.0'
|
|
35
|
+
spec.add_dependency 'fugit', '~> 1.0'
|
|
36
|
+
|
|
37
|
+
# Optional: add to your own Gemfile if you need these features
|
|
38
|
+
# gem 'sqlite3', '~> 2.0' # dynamic job queue
|
|
39
|
+
# gem 'async-utilization', '~> 0.3' # shared-memory worker metrics
|
|
40
|
+
|
|
41
|
+
spec.add_development_dependency 'rake', '~> 13.0'
|
|
42
|
+
spec.add_development_dependency 'rspec', '~> 3.12'
|
|
43
|
+
end
|
|
@@ -4,7 +4,19 @@ module Async
|
|
|
4
4
|
module Background
|
|
5
5
|
module Queue
|
|
6
6
|
class Notifier
|
|
7
|
-
|
|
7
|
+
# Error groups
|
|
8
|
+
WRITE_DROPPED = [
|
|
9
|
+
IO::WaitWritable, # buffer full — consumer is behind, skip
|
|
10
|
+
Errno::EAGAIN, # same as above on some platforms
|
|
11
|
+
IOError, # our own writer end has been closed
|
|
12
|
+
Errno::EPIPE # the reader end is gone (consumer crashed) — skip
|
|
13
|
+
].freeze
|
|
14
|
+
|
|
15
|
+
READ_EXHAUSTED = [
|
|
16
|
+
IO::WaitReadable, # nothing left in the buffer — normal exit
|
|
17
|
+
EOFError, # writer end closed — no more data ever
|
|
18
|
+
IOError # our own reader end has been closed
|
|
19
|
+
].freeze
|
|
8
20
|
|
|
9
21
|
attr_reader :reader, :writer
|
|
10
22
|
|
|
@@ -16,10 +28,10 @@ module Async
|
|
|
16
28
|
|
|
17
29
|
def notify
|
|
18
30
|
@writer.write_nonblock("\x01")
|
|
19
|
-
rescue
|
|
20
|
-
#
|
|
21
|
-
|
|
22
|
-
#
|
|
31
|
+
rescue *WRITE_DROPPED
|
|
32
|
+
# All write failures are non-fatal: the job is already in the
|
|
33
|
+
# store, and missing one wake-up only delays pickup by at most
|
|
34
|
+
# one poll interval.
|
|
23
35
|
end
|
|
24
36
|
|
|
25
37
|
def wait(timeout: nil)
|
|
@@ -53,7 +65,7 @@ module Async
|
|
|
53
65
|
def drain
|
|
54
66
|
loop do
|
|
55
67
|
@reader.read_nonblock(256)
|
|
56
|
-
rescue *
|
|
68
|
+
rescue *READ_EXHAUSTED
|
|
57
69
|
break
|
|
58
70
|
end
|
|
59
71
|
nil
|
|
@@ -53,6 +53,7 @@ module Async
|
|
|
53
53
|
def ensure_database!
|
|
54
54
|
require_sqlite3
|
|
55
55
|
db = SQLite3::Database.new(@path)
|
|
56
|
+
db.execute('PRAGMA busy_timeout = 5000')
|
|
56
57
|
db.execute_batch(PRAGMAS.call(@mmap ? MMAP_SIZE : 0))
|
|
57
58
|
db.execute_batch(SCHEMA)
|
|
58
59
|
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
|
@@ -128,6 +129,7 @@ module Async
|
|
|
128
129
|
require_sqlite3
|
|
129
130
|
finalize_statements
|
|
130
131
|
@db = SQLite3::Database.new(@path)
|
|
132
|
+
@db.execute('PRAGMA busy_timeout = 5000')
|
|
131
133
|
@db.execute_batch(PRAGMAS.call(@mmap ? MMAP_SIZE : 0))
|
|
132
134
|
|
|
133
135
|
unless @schema_checked
|
data/lib/async/background.rb
CHANGED
|
@@ -12,5 +12,6 @@ require_relative 'background/min_heap'
|
|
|
12
12
|
require_relative 'background/entry'
|
|
13
13
|
require_relative 'background/metrics'
|
|
14
14
|
require_relative 'background/runner'
|
|
15
|
+
require_relative 'background/queue/store'
|
|
15
16
|
require_relative 'background/queue/client'
|
|
16
17
|
require_relative 'background/job'
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: async-background
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.5.
|
|
4
|
+
version: 0.5.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Roman Hajdarov
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-04-07 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: async
|
|
@@ -90,6 +90,10 @@ executables: []
|
|
|
90
90
|
extensions: []
|
|
91
91
|
extra_rdoc_files: []
|
|
92
92
|
files:
|
|
93
|
+
- CHANGELOG.md
|
|
94
|
+
- LICENSE
|
|
95
|
+
- README.md
|
|
96
|
+
- async-background.gemspec
|
|
93
97
|
- lib/async/background.rb
|
|
94
98
|
- lib/async/background/clock.rb
|
|
95
99
|
- lib/async/background/entry.rb
|