monkrb 0.17.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +56 -0
- data/README.md +5 -4
- data/exe/monk +45 -7
- data/lib/monk/jobs/adapters/pg.rb +272 -0
- data/lib/monk/jobs/args.rb +43 -0
- data/lib/monk/jobs/claim.rb +7 -0
- data/lib/monk/jobs/errors.rb +29 -0
- data/lib/monk/jobs/job.rb +117 -0
- data/lib/monk/jobs/runtime.rb +239 -0
- data/lib/monk/jobs/worker.rb +94 -0
- data/lib/monk/jobs.rb +229 -0
- data/lib/monk/mail/errors.rb +7 -0
- data/lib/monk/mail/later.rb +53 -0
- data/lib/monk/persistence/pg.rb +21 -1
- data/lib/monk/scaffold.rb +232 -8
- data/lib/monk/templates/jobs/bin/jobs +26 -0
- data/lib/monk/templates/jobs/config/jobs.rb +17 -0
- data/lib/monk/templates/jobs/db/migrate/00000000000002_create_jobs_tables.down.sql +3 -0
- data/lib/monk/templates/jobs/db/migrate/00000000000002_create_jobs_tables.up.sql +48 -0
- data/lib/monk/templates/jobs/jobs/hello_job.rb +9 -0
- data/lib/monk/templates/jobs/jobs/send_login_link.rb +27 -0
- data/lib/monk/templates/postgres/Dockerfile +3 -1
- data/lib/monk/version.rb +1 -1
- metadata +16 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e8e07694f006907b83202dbafa0dcbfd61d31e1c1f90b1012597dada8eb24c3e
|
|
4
|
+
data.tar.gz: cbadf97dbb1fb1063332c7046afcc96fd526e7c483c0bcc8a5873f1b94bb2636
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f44edaf6f9f80765fa51841d068b920f880fc6855fd702cbff242a51423ca4f7206452b9f4899a6ca5340b2ac228fd8604c287e1dad82d9f122ea6ac26304b2d
|
|
7
|
+
data.tar.gz: 1dc1579210fbbd2c211a590eb50ccabf9c385b7e2d8362b723c0cbe1edd55ad64bfd886d8948725d4514b84afee274144fd25790dbf554b82a0372072319e33d
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,62 @@ All notable changes to this project are documented here. Format is loosely
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/); versions are as released
|
|
5
5
|
in `lib/monk/version.rb`.
|
|
6
6
|
|
|
7
|
+
## 0.18.0 - 2026-09-29
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- `Monk::Jobs` (`require "monk/jobs"`): background jobs on Postgres, with
|
|
12
|
+
no other dependency. A job is a `Monk::Job` subclass with
|
|
13
|
+
`self.perform` and optional `queue`, `priority`, `max_attempts` and
|
|
14
|
+
`timeout` settings, enqueued with `SendReceipt.enqueue(*args, wait:,
|
|
15
|
+
at:, conn:)`. Args must be plain JSON values, checked at enqueue.
|
|
16
|
+
`conn:` enqueues inside the app's own transaction. The queue is a narrow
|
|
17
|
+
state table plus a payload table, claimed with `FOR UPDATE SKIP
|
|
18
|
+
LOCKED`: ADR 0013 records the design and the benchmarks that chose it
|
|
19
|
+
over one wide table and over Solid Queue's split.
|
|
20
|
+
`Monk::Jobs::Runtime` (`require "monk/jobs/runtime"`) is the job
|
|
21
|
+
process. It runs a supervisor and a pool of worker Ractors, retries
|
|
22
|
+
failures with backoff, keeps jobs that run out of attempts as failed
|
|
23
|
+
(`Monk::Jobs.retry_failed`/`discard_failed`), and releases the jobs of
|
|
24
|
+
dead workers and dead processes. On `TERM` it finishes the jobs in hand
|
|
25
|
+
first. Delivery is at-least-once. For tests, `Monk::Jobs.drain!` runs
|
|
26
|
+
every due job in its own Ractor, and `Monk::Jobs.clear!` empties the
|
|
27
|
+
queue (test environment only). See `docs/guides/jobs.md`.
|
|
28
|
+
- `monk new --jobs` (implies `--postgres`): `config/jobs.rb`, a demo
|
|
29
|
+
`HelloJob` with a `POST /jobs/hello` route, `bin/jobs`, the queue's
|
|
30
|
+
migration, `JOBS_WORKERS`/`JOBS_QUEUES` in `.env`, and `SETUP.md`
|
|
31
|
+
steps for running and testing jobs.
|
|
32
|
+
- `Monk::Mail.deliver_later` (`require "monk/mail/later"`): `deliver`'s
|
|
33
|
+
arguments plus `wait:`/`at:`/`conn:`, sent from a job on the `mailers`
|
|
34
|
+
queue. The message is checked when called, so bad input raises there.
|
|
35
|
+
Temporary failures are retried, while a refusal the server made final
|
|
36
|
+
(SMTP 5xx, refused credentials) fails the job at once as the new
|
|
37
|
+
`Monk::Mail::PermanentDeliveryError`, a `DeliveryError`. `deliver` is
|
|
38
|
+
unchanged. See ADR 0014.
|
|
39
|
+
- `never_retry *error_classes` on `Monk::Job`: errors that fail a job at
|
|
40
|
+
once instead of spending its remaining attempts.
|
|
41
|
+
- `monk new` prints the flags it resolved: which ones another flag turned
|
|
42
|
+
on, and why (`--postgres (needed by --auth, --jobs)`), and which pairs
|
|
43
|
+
change what gets generated. `monk --help` and the scaffolding guide
|
|
44
|
+
state the rule behind it: a flag is implied when there's only one right
|
|
45
|
+
answer, and required when there's a real choice (`--live`'s transport).
|
|
46
|
+
- `monk new --auth --jobs` adds `jobs/send_login_link.rb`, which creates
|
|
47
|
+
the login token and sends the link inside the job, so the raw token is
|
|
48
|
+
never stored in the queue. With mail, `--jobs` also loads
|
|
49
|
+
`deliver_later` and serves the `mailers` queue first.
|
|
50
|
+
|
|
51
|
+
### Fixed
|
|
52
|
+
|
|
53
|
+
- `json` and `jsonb` columns read through `Monk::Persistence::Pg` (raw
|
|
54
|
+
queries and `Model` alike) raised `ArgumentError: unknown keyword:
|
|
55
|
+
quirks_mode`. pg 1.6.3's JSON decoder passes that keyword to
|
|
56
|
+
`JSON.parse`, and json 3 removed it. Monk's connections now decode
|
|
57
|
+
`json`/`jsonb` with their own decoder, keeping pg's other default
|
|
58
|
+
types.
|
|
59
|
+
- The sample test `SETUP.md` gives a `monk new --postgres` app (without
|
|
60
|
+
`--auth`) failed as written: it expected `SELECT 1` to return the
|
|
61
|
+
String `"1"`, but Monk's connections decode it to the Integer `1`.
|
|
62
|
+
|
|
7
63
|
## 0.17.0 - 2026-09-26
|
|
8
64
|
|
|
9
65
|
### Added
|
data/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
A light Ruby web framework designed to be fully `Ractor`-safe: every app it produces is a valid Rack 3 app that is also `Ractor.shareable?`, so it can be served in parallel across Ractor worker pools without silently losing that safety property. Named after Thelonious Sphere Monk, great and unique Jazz piano player and composer.
|
|
4
4
|
|
|
5
|
-
**Built in** (loaded by `require "monk"`): routing, context and error handling, boot-time Ractor-shareability checks, `Monk::StateRactor` for shared state, settings, ERB views, static assets and logging. **Opt-in** (each needs its own `require`): Postgres persistence and migrations, passwordless auth and sessions, a WebSocket server with Redis or Postgres fan-out, and `Monk::Live` server-pushed HTML updates. Details for each are in the [Features](#features) table below.
|
|
5
|
+
**Built in** (loaded by `require "monk"`): routing, context and error handling, boot-time Ractor-shareability checks, `Monk::StateRactor` for shared state, settings, ERB views, static assets and logging. **Opt-in** (each needs its own `require`): Postgres persistence and migrations, passwordless auth and sessions, email, background jobs on Postgres, a WebSocket server with Redis or Postgres fan-out, and `Monk::Live` server-pushed HTML updates. Details for each are in the [Features](#features) table below.
|
|
6
6
|
|
|
7
7
|
Monk is Kino-agnostic — it's built on stdlib `Ractor` primitives only, with no runtime dependency on any particular server. [Kino](https://github.com/yaroslav/kino) is the reference/development server (see `bin/server`), but any Ractor-aware Rack server, or a conventional one, can run a Monk app.
|
|
8
8
|
|
|
@@ -22,7 +22,7 @@ bin/server # -> http://localhost:9292/hello
|
|
|
22
22
|
|
|
23
23
|
`monk new` scaffolds the app's own `Gemfile` with `gem "monkrb", require: "monk"`.
|
|
24
24
|
|
|
25
|
-
`monk new` writes a working skeleton (an HTML home page, a `/hello` route, a `/api/hello` JSON route, `views/`, `public/`, a `SETUP.md`). Flags add Postgres, auth, Redis and live updates: see [`docs/guides/scaffolding.md`](docs/guides/scaffolding.md).
|
|
25
|
+
`monk new` writes a working skeleton (an HTML home page, a `/hello` route, a `/api/hello` JSON route, `views/`, `public/`, a `SETUP.md`). Flags add Postgres, auth, email, background jobs, Redis and live updates: see [`docs/guides/scaffolding.md`](docs/guides/scaffolding.md).
|
|
26
26
|
|
|
27
27
|
All `monk` commands and flags are listed by:
|
|
28
28
|
|
|
@@ -60,10 +60,11 @@ Everything beyond the core is opt-in (`require "monk"` alone loads none of it).
|
|
|
60
60
|
| Persistence | `Monk::Persistence::Pg`: raw `pg`, per-Ractor connections, hash-based `Model` | `require "monk/persistence/pg"` (+ `.../pg/model`); needs the `pg` gem | [`persistence.md`](docs/guides/persistence.md) |
|
|
61
61
|
| Migrations | plain `.sql` up/down pairs, `Migrator` | `require "monk/persistence/pg/migrator"`; needs the `pg` gem | [`migrations.md`](docs/guides/migrations.md) |
|
|
62
62
|
| Auth and sessions | `Monk::Auth`: passwordless tokens, Bearer or cookie + CSRF | `require "monk/auth"`; needs the `pg` gem and a registered Postgres connection | [`auth.md`](docs/guides/auth.md) |
|
|
63
|
-
| Email | `Monk::Mail`: text/HTML email over SMTP or a local relay, one `MAIL_URL`, sent from any worker Ractor | `require "monk/mail"
|
|
63
|
+
| Email | `Monk::Mail`: text/HTML email over SMTP or a local relay, one `MAIL_URL`, sent from any worker Ractor, or from a background job with `deliver_later` | `require "monk/mail"` (+ `require "monk/mail/later"` for `deliver_later`, which needs background jobs); SMTP needs the `net-smtp` gem | [`mail.md`](docs/guides/mail.md) |
|
|
64
|
+
| Background jobs | `Monk::Jobs`: a queue in Postgres, run by `bin/jobs` on worker Ractors; retries with backoff, scheduled jobs, enqueue inside the app's own transaction, `drain!` for tests | `require "monk/jobs"` (+ `require "monk/jobs/runtime"` in the job process); needs the `pg` gem and a registered Postgres connection | [`jobs.md`](docs/guides/jobs.md) |
|
|
64
65
|
| WebSocket | `Monk::WebSocket`: RFC 6455 server as its own process, Redis or Postgres fan-out | `require "monk/websocket"`; fan-out: `require "monk/websocket/redis_fanout"` (+ the `redis` gem) or `require "monk/websocket/pg_fanout"` (+ the `pg` gem) | [`websocket.md`](docs/guides/websocket.md) |
|
|
65
66
|
| Live updates | `Monk::Live`: server-rendered HTML patches pushed to open tabs | `require "monk/live"`; needs the WebSocket server and, across processes, Redis or Postgres | [`live.md`](docs/guides/live.md) |
|
|
66
|
-
| Scaffolding | `monk new` and its flags; retrofitting Postgres, Auth or
|
|
67
|
+
| Scaffolding | `monk new` and its flags; retrofitting Postgres, Auth, Redis or Jobs | — (the `monk` command; flags `--postgres`, `--auth`, `--mail`, `--jobs`, `--redis`, `--live`) | [`scaffolding.md`](docs/guides/scaffolding.md) |
|
|
67
68
|
|
|
68
69
|
## More documentation
|
|
69
70
|
|
data/exe/monk
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env ruby
|
|
2
2
|
require_relative "../lib/monk/scaffold"
|
|
3
3
|
|
|
4
|
-
USAGE = "Usage: monk new APP_NAME [--postgres] [--auth] [--mail] [--redis] [--live]".freeze
|
|
4
|
+
USAGE = "Usage: monk new APP_NAME [--postgres] [--auth] [--mail] [--redis] [--live] [--jobs]".freeze
|
|
5
5
|
|
|
6
6
|
HELP = <<~TEXT.freeze
|
|
7
7
|
#{USAGE}
|
|
@@ -79,6 +79,37 @@ HELP = <<~TEXT.freeze
|
|
|
79
79
|
views/index.erb and bin/websocket_server with live
|
|
80
80
|
versions, and adds the runtime's <meta>/<script> tags
|
|
81
81
|
to the layout.
|
|
82
|
+
--jobs Also scaffold Monk::Jobs (background jobs on Postgres).
|
|
83
|
+
Implies --postgres, where the queue lives. Adds
|
|
84
|
+
config/jobs.rb (required by config.ru, after
|
|
85
|
+
persistence/auth/mail), jobs/hello_job.rb (a demo
|
|
86
|
+
job, enqueued by a POST /jobs/hello route added to
|
|
87
|
+
config.ru), bin/jobs (the job process, run beside
|
|
88
|
+
bin/server), and the migration creating the queue's
|
|
89
|
+
tables. .env/.env.example get JOBS_WORKERS and
|
|
90
|
+
JOBS_QUEUES (not .env.test: tests run jobs with
|
|
91
|
+
Monk::Jobs.drain!).
|
|
92
|
+
|
|
93
|
+
How flags combine -- a flag is implied when there's only one right
|
|
94
|
+
answer, and required when there's a real choice:
|
|
95
|
+
|
|
96
|
+
Flag Implies Requires
|
|
97
|
+
--auth --postgres, --mail -
|
|
98
|
+
--jobs --postgres -
|
|
99
|
+
--live - --redis or --postgres (Live's
|
|
100
|
+
transport: a choice, so it's yours)
|
|
101
|
+
--postgres, --mail, --redis stand alone.
|
|
102
|
+
|
|
103
|
+
Some pairs also change what gets generated:
|
|
104
|
+
|
|
105
|
+
--auth + --jobs jobs/send_login_link.rb: login links sent from a job
|
|
106
|
+
--mail + --jobs config/jobs.rb loads Monk::Mail.deliver_later;
|
|
107
|
+
JOBS_QUEUES serves mailers first
|
|
108
|
+
--live + --redis config/live.rb fans out over Redis
|
|
109
|
+
--live + --postgres config/live.rb fans out over Postgres (without --redis)
|
|
110
|
+
|
|
111
|
+
`monk new` prints which flags it turned on, and why, after creating the
|
|
112
|
+
project.
|
|
82
113
|
|
|
83
114
|
Commands:
|
|
84
115
|
new APP_NAME Create a new project (see Options above)
|
|
@@ -119,20 +150,27 @@ when "new"
|
|
|
119
150
|
app_name = rest.find { |arg| !arg.start_with?("-") }
|
|
120
151
|
usage_error! if app_name.nil?
|
|
121
152
|
|
|
153
|
+
# Flags as typed: Monk::Scaffold resolves what they imply, and reports it.
|
|
122
154
|
auth = rest.include?("--auth")
|
|
123
|
-
mail = rest.include?("--mail") || auth
|
|
155
|
+
mail = rest.include?("--mail") || auth # for the next-steps lines below
|
|
124
156
|
redis = rest.include?("--redis")
|
|
125
157
|
live = rest.include?("--live")
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
158
|
+
jobs = rest.include?("--jobs")
|
|
159
|
+
scaffold = Monk::Scaffold.new(
|
|
160
|
+
app_name, postgres: rest.include?("--postgres"), auth: auth, mail: rest.include?("--mail"), redis: redis,
|
|
161
|
+
live: live, jobs: jobs,
|
|
162
|
+
)
|
|
163
|
+
scaffold.write!
|
|
164
|
+
|
|
165
|
+
puts "Created #{app_name}."
|
|
166
|
+
puts scaffold.summary
|
|
167
|
+
puts "Next steps:"
|
|
131
168
|
puts " cd #{app_name}"
|
|
132
169
|
puts " bundle install"
|
|
133
170
|
puts " see SETUP.md for the full dev-then-test walkthrough"
|
|
134
171
|
puts " change the placeholder AUTH_SECRET in .env/.env.test before relying on it" if auth
|
|
135
172
|
puts " set MAIL_URL and MAIL_FROM outside development (see config/mail.rb)" if mail
|
|
173
|
+
puts " run bin/jobs beside bin/server to process background jobs (see SETUP.md)" if jobs
|
|
136
174
|
if live
|
|
137
175
|
if redis
|
|
138
176
|
puts " Monk::Live needs Redis running, bin/server and bin/websocket_server (see SETUP.md)"
|
|
@@ -0,0 +1,272 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
require "time"
|
|
3
|
+
require_relative "../../persistence/pg"
|
|
4
|
+
require_relative "../claim"
|
|
5
|
+
|
|
6
|
+
module Monk
|
|
7
|
+
module Jobs
|
|
8
|
+
module Adapters
|
|
9
|
+
# The queue on Postgres: monk_jobs + monk_job_payloads + monk_processes
|
|
10
|
+
# (docs/adr/0013-jobs-narrow-state-table-plus-payloads.md). Holds only
|
|
11
|
+
# the registered database's name and freezes itself, so the one
|
|
12
|
+
# instance is shareable with every Ractor; each Ractor reaches the
|
|
13
|
+
# database through its own connection, via Monk::Persistence::Pg.
|
|
14
|
+
class Pg
|
|
15
|
+
# One statement, so a job and its payload are written together even
|
|
16
|
+
# outside a transaction. The database clock decides run_at, and so
|
|
17
|
+
# whether the job starts scheduled or available -- an app server's
|
|
18
|
+
# clock running a little ahead can't make a job look due early.
|
|
19
|
+
ENQUEUE_SQL = <<~SQL.freeze
|
|
20
|
+
WITH due AS (
|
|
21
|
+
SELECT COALESCE($4::timestamptz, now() + COALESCE($5::float8, 0) * interval '1 second') AS run_at
|
|
22
|
+
), job AS (
|
|
23
|
+
INSERT INTO monk_jobs (queue, priority, max_attempts, run_at, state)
|
|
24
|
+
SELECT $1::text, $2::smallint, $3::smallint, run_at,
|
|
25
|
+
CASE WHEN run_at > now() THEN 'scheduled' ELSE 'available' END
|
|
26
|
+
FROM due
|
|
27
|
+
RETURNING id
|
|
28
|
+
)
|
|
29
|
+
INSERT INTO monk_job_payloads (job_id, job_class, args)
|
|
30
|
+
SELECT id, $6::text, $7::jsonb FROM job
|
|
31
|
+
RETURNING job_id
|
|
32
|
+
SQL
|
|
33
|
+
|
|
34
|
+
# Decision 2: the claim rewrites only the narrow monk_jobs row. args
|
|
35
|
+
# comes back as text and is parsed here rather than by the
|
|
36
|
+
# connection's result type map, so it decodes the same whichever
|
|
37
|
+
# connection runs it.
|
|
38
|
+
CLAIM_SQL = <<~SQL.freeze
|
|
39
|
+
WITH claimed AS (
|
|
40
|
+
UPDATE monk_jobs
|
|
41
|
+
SET state = 'running', locked_by = $2, locked_at = now(), attempts = attempts + 1
|
|
42
|
+
WHERE id = (
|
|
43
|
+
SELECT id FROM monk_jobs
|
|
44
|
+
WHERE queue = $1 AND state = 'available'
|
|
45
|
+
ORDER BY priority, run_at, id
|
|
46
|
+
FOR UPDATE SKIP LOCKED
|
|
47
|
+
LIMIT 1
|
|
48
|
+
)
|
|
49
|
+
RETURNING id, attempts
|
|
50
|
+
)
|
|
51
|
+
SELECT claimed.id, claimed.attempts, payload.job_class, payload.args::text AS args
|
|
52
|
+
FROM claimed JOIN monk_job_payloads payload ON payload.job_id = claimed.id
|
|
53
|
+
SQL
|
|
54
|
+
|
|
55
|
+
# Only while this process still holds the job: one pruned from a dead
|
|
56
|
+
# process and claimed again elsewhere isn't deleted by the first
|
|
57
|
+
# worker finishing late. The payload goes by ON DELETE CASCADE.
|
|
58
|
+
FINISH_SQL = <<~SQL.freeze
|
|
59
|
+
DELETE FROM monk_jobs WHERE id = $1 AND state = 'running' AND locked_by = $2
|
|
60
|
+
SQL
|
|
61
|
+
|
|
62
|
+
# Only while this process still holds the job, like FINISH_SQL. The
|
|
63
|
+
# database decides where it goes: back to scheduled, due retry_in
|
|
64
|
+
# seconds from now, or to failed once it has used its last attempt
|
|
65
|
+
# (attempts was already counted when it was claimed) or when
|
|
66
|
+
# retry_in is NULL. The payload keeps the error either way; the
|
|
67
|
+
# payload update runs whether or not the final SELECT reads it.
|
|
68
|
+
FAIL_SQL = <<~SQL.freeze
|
|
69
|
+
WITH job AS (
|
|
70
|
+
UPDATE monk_jobs SET
|
|
71
|
+
state = CASE WHEN $3::float8 IS NULL OR attempts >= max_attempts THEN 'failed' ELSE 'scheduled' END,
|
|
72
|
+
run_at = CASE WHEN $3::float8 IS NULL OR attempts >= max_attempts THEN run_at
|
|
73
|
+
ELSE now() + $3::float8 * interval '1 second' END,
|
|
74
|
+
locked_by = NULL, locked_at = NULL
|
|
75
|
+
WHERE id = $1 AND state = 'running' AND locked_by = $2
|
|
76
|
+
RETURNING id, state
|
|
77
|
+
), payload AS (
|
|
78
|
+
UPDATE monk_job_payloads SET last_error = $4 FROM job WHERE monk_job_payloads.job_id = job.id
|
|
79
|
+
)
|
|
80
|
+
SELECT state FROM job
|
|
81
|
+
SQL
|
|
82
|
+
|
|
83
|
+
# The stager: due scheduled jobs become available, oldest first, at
|
|
84
|
+
# most $1 per call. SKIP LOCKED lets every job process stage at
|
|
85
|
+
# once without two of them moving the same job.
|
|
86
|
+
STAGE_SQL = <<~SQL.freeze
|
|
87
|
+
UPDATE monk_jobs SET state = 'available'
|
|
88
|
+
WHERE id IN (
|
|
89
|
+
SELECT id FROM monk_jobs
|
|
90
|
+
WHERE state = 'scheduled' AND run_at <= now()
|
|
91
|
+
ORDER BY run_at
|
|
92
|
+
LIMIT $1
|
|
93
|
+
FOR UPDATE SKIP LOCKED
|
|
94
|
+
)
|
|
95
|
+
SQL
|
|
96
|
+
|
|
97
|
+
RETRY_FAILED_SQL = <<~SQL.freeze
|
|
98
|
+
UPDATE monk_jobs SET state = 'available', attempts = 0, run_at = now() WHERE id = $1 AND state = 'failed'
|
|
99
|
+
SQL
|
|
100
|
+
|
|
101
|
+
DISCARD_FAILED_SQL = <<~SQL.freeze
|
|
102
|
+
DELETE FROM monk_jobs WHERE id = $1 AND state = 'failed'
|
|
103
|
+
SQL
|
|
104
|
+
|
|
105
|
+
REGISTER_PROCESS_SQL = <<~SQL.freeze
|
|
106
|
+
INSERT INTO monk_processes (hostname, pid) VALUES ($1, $2) RETURNING id
|
|
107
|
+
SQL
|
|
108
|
+
|
|
109
|
+
# An upsert, not an UPDATE: a process that stalled long enough for
|
|
110
|
+
# another to prune it gets its row back under the same id. The jobs
|
|
111
|
+
# pruning released stay with whoever claimed them since; this
|
|
112
|
+
# process's late finish/fail calls on them just find nothing held.
|
|
113
|
+
HEARTBEAT_SQL = <<~SQL.freeze
|
|
114
|
+
INSERT INTO monk_processes (id, hostname, pid) OVERRIDING SYSTEM VALUE VALUES ($1, $2, $3)
|
|
115
|
+
ON CONFLICT (id) DO UPDATE SET last_heartbeat_at = now()
|
|
116
|
+
SQL
|
|
117
|
+
|
|
118
|
+
# Deletes processes whose heartbeat is older than $1 seconds, and
|
|
119
|
+
# releases their running jobs back to available with their attempt
|
|
120
|
+
# still counted (Decision 4). Also releases running jobs with no
|
|
121
|
+
# process row at all once they've been running that long -- a
|
|
122
|
+
# drain! that crashed mid-job, or rows left by a bug. The NOT EXISTS
|
|
123
|
+
# sees the rows as they were before the DELETE, so a pruned
|
|
124
|
+
# process's jobs are caught by the first condition.
|
|
125
|
+
PRUNE_SQL = <<~SQL.freeze
|
|
126
|
+
WITH dead AS (
|
|
127
|
+
DELETE FROM monk_processes
|
|
128
|
+
WHERE last_heartbeat_at < now() - $1::float8 * interval '1 second'
|
|
129
|
+
RETURNING id
|
|
130
|
+
)
|
|
131
|
+
UPDATE monk_jobs SET state = 'available', locked_by = NULL, locked_at = NULL
|
|
132
|
+
WHERE state = 'running' AND (
|
|
133
|
+
locked_by IN (SELECT id FROM dead)
|
|
134
|
+
OR (locked_at < now() - $1::float8 * interval '1 second'
|
|
135
|
+
AND NOT EXISTS (SELECT 1 FROM monk_processes p WHERE p.id = monk_jobs.locked_by))
|
|
136
|
+
)
|
|
137
|
+
SQL
|
|
138
|
+
|
|
139
|
+
# A graceful stop: whatever this process still has running goes back
|
|
140
|
+
# to available, and its row goes.
|
|
141
|
+
DEREGISTER_SQL = <<~SQL.freeze
|
|
142
|
+
WITH gone AS (DELETE FROM monk_processes WHERE id = $1)
|
|
143
|
+
UPDATE monk_jobs SET state = 'available', locked_by = NULL, locked_at = NULL
|
|
144
|
+
WHERE state = 'running' AND locked_by = $1
|
|
145
|
+
SQL
|
|
146
|
+
|
|
147
|
+
# One job back to available, if process_id still holds it: the job a
|
|
148
|
+
# worker was running when that worker died.
|
|
149
|
+
RELEASE_SQL = <<~SQL.freeze
|
|
150
|
+
UPDATE monk_jobs SET state = 'available', locked_by = NULL, locked_at = NULL
|
|
151
|
+
WHERE id = $1 AND state = 'running' AND locked_by = $2
|
|
152
|
+
SQL
|
|
153
|
+
|
|
154
|
+
CLEAR_SQL = <<~SQL.freeze
|
|
155
|
+
DELETE FROM monk_jobs;
|
|
156
|
+
DELETE FROM monk_processes;
|
|
157
|
+
SQL
|
|
158
|
+
|
|
159
|
+
attr_reader :db_name
|
|
160
|
+
|
|
161
|
+
def initialize(db_name:)
|
|
162
|
+
@db_name = db_name
|
|
163
|
+
freeze
|
|
164
|
+
end
|
|
165
|
+
|
|
166
|
+
# Returns the new job's id. conn: runs it on a connection the caller
|
|
167
|
+
# already holds, inside the caller's transaction -- required rather
|
|
168
|
+
# than optional there, since checking out the same Ractor's
|
|
169
|
+
# connection again would wait on itself.
|
|
170
|
+
def enqueue(job_class:, queue:, priority:, max_attempts:, args:, wait: nil, at: nil, conn: nil)
|
|
171
|
+
params = [queue, priority, max_attempts, at&.utc&.iso8601(6), wait&.to_f, job_class, args]
|
|
172
|
+
with_connection(conn) { |c| Integer(c.exec_params(ENQUEUE_SQL, params).getvalue(0, 0)) }
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
# The next available job on `queue`, now running for process_id, or
|
|
176
|
+
# nil when there's none.
|
|
177
|
+
def claim(queue, process_id)
|
|
178
|
+
row = with_connection { |c| c.exec_params(CLAIM_SQL, [queue, process_id]).first }
|
|
179
|
+
return nil unless row
|
|
180
|
+
|
|
181
|
+
Claim.new(
|
|
182
|
+
id: Integer(row["id"]), job_class: row["job_class"],
|
|
183
|
+
args: JSON.parse(row["args"]), attempts: Integer(row["attempts"]),
|
|
184
|
+
)
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# True if the job was deleted, false if process_id no longer held it.
|
|
188
|
+
def finish(id, process_id)
|
|
189
|
+
with_connection { |c| c.exec_params(FINISH_SQL, [id, process_id]).cmd_tuples == 1 }
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
# Records a failure of a job process_id holds: :scheduled when it will
|
|
193
|
+
# be retried in retry_in seconds, :failed when it won't (no attempts
|
|
194
|
+
# left, or retry_in nil), nil when process_id no longer held it.
|
|
195
|
+
def fail(id, process_id, error:, retry_in:)
|
|
196
|
+
row = with_connection { |c| c.exec_params(FAIL_SQL, [id, process_id, retry_in&.to_f, error]).first }
|
|
197
|
+
row && row["state"].to_sym
|
|
198
|
+
end
|
|
199
|
+
|
|
200
|
+
# Moves up to `limit` due scheduled jobs to available; returns how many.
|
|
201
|
+
def stage_due(limit = 500)
|
|
202
|
+
with_connection { |c| c.exec_params(STAGE_SQL, [limit]).cmd_tuples }
|
|
203
|
+
end
|
|
204
|
+
|
|
205
|
+
# A failed job back to available with fresh attempts. False if it
|
|
206
|
+
# isn't failed (or doesn't exist).
|
|
207
|
+
def retry_failed(id)
|
|
208
|
+
with_connection { |c| c.exec_params(RETRY_FAILED_SQL, [id]).cmd_tuples == 1 }
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
# Deletes a failed job and its payload. False if it isn't failed.
|
|
212
|
+
def discard_failed(id)
|
|
213
|
+
with_connection { |c| c.exec_params(DISCARD_FAILED_SQL, [id]).cmd_tuples == 1 }
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
# A job process's own row; returns its id, the process_id it claims
|
|
217
|
+
# jobs as.
|
|
218
|
+
def register_process(hostname:, pid:)
|
|
219
|
+
with_connection { |c| Integer(c.exec_params(REGISTER_PROCESS_SQL, [hostname, pid]).getvalue(0, 0)) }
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def heartbeat(process_id, hostname:, pid:)
|
|
223
|
+
with_connection { |c| c.exec_params(HEARTBEAT_SQL, [process_id, hostname, pid]) }
|
|
224
|
+
nil
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
# Releases the jobs of processes silent for more than `timeout`
|
|
228
|
+
# seconds, and deletes those processes; returns how many jobs were
|
|
229
|
+
# released.
|
|
230
|
+
def prune(timeout)
|
|
231
|
+
with_connection { |c| c.exec_params(PRUNE_SQL, [timeout.to_f]).cmd_tuples }
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
# The job a dead worker was holding, back to available with its
|
|
235
|
+
# attempt counted; true if process_id still held it.
|
|
236
|
+
def release(id, process_id)
|
|
237
|
+
with_connection { |c| c.exec_params(RELEASE_SQL, [id, process_id]).cmd_tuples == 1 }
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
# Releases this process's running jobs and deletes its row.
|
|
241
|
+
def deregister(process_id)
|
|
242
|
+
with_connection { |c| c.exec_params(DEREGISTER_SQL, [process_id]) }
|
|
243
|
+
nil
|
|
244
|
+
end
|
|
245
|
+
|
|
246
|
+
# After a job timed out mid-query: cancels whatever the calling
|
|
247
|
+
# Ractor's connection is still running on the server, then
|
|
248
|
+
# reconnects, so the worker's next query doesn't wait for the
|
|
249
|
+
# abandoned one (Phase 0.3) or land inside its open transaction.
|
|
250
|
+
def reset_connection
|
|
251
|
+
conn = Monk::Persistence::Pg[@db_name]
|
|
252
|
+
conn.cancel
|
|
253
|
+
conn.reset
|
|
254
|
+
nil
|
|
255
|
+
end
|
|
256
|
+
|
|
257
|
+
# Every job (payloads by cascade) and every process row. Behind
|
|
258
|
+
# Monk::Jobs.clear!, which only runs in tests.
|
|
259
|
+
def clear!
|
|
260
|
+
with_connection { |c| c.exec(CLEAR_SQL) }
|
|
261
|
+
nil
|
|
262
|
+
end
|
|
263
|
+
|
|
264
|
+
private
|
|
265
|
+
|
|
266
|
+
def with_connection(conn = nil, &)
|
|
267
|
+
conn ? yield(conn) : Monk::Persistence::Pg.checkout(@db_name, &)
|
|
268
|
+
end
|
|
269
|
+
end
|
|
270
|
+
end
|
|
271
|
+
end
|
|
272
|
+
end
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
module Monk
|
|
2
|
+
module Jobs
|
|
3
|
+
# A job's args are stored as JSON (monk_job_payloads.args) and handed
|
|
4
|
+
# back to #perform in another process, so only values that come back
|
|
5
|
+
# from that round trip unchanged are accepted: Strings, Integers,
|
|
6
|
+
# finite Floats, true/false/nil, and Arrays and String-keyed Hashes of
|
|
7
|
+
# those. Checked when a job is enqueued, so a bad argument fails in
|
|
8
|
+
# the code that passed it rather than later, in a worker.
|
|
9
|
+
module Args
|
|
10
|
+
def self.check!(args)
|
|
11
|
+
raise InvalidArgumentsError, "a job's args must be an Array, got #{args.class}" unless args.is_a?(Array)
|
|
12
|
+
|
|
13
|
+
args.each_with_index { |value, i| check_value!(value, "args[#{i}]") }
|
|
14
|
+
nil
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def self.check_value!(value, path)
|
|
18
|
+
case value
|
|
19
|
+
when String, Integer, true, false, nil
|
|
20
|
+
nil
|
|
21
|
+
when Float
|
|
22
|
+
raise InvalidArgumentsError, "#{path} is #{value}, which JSON can't represent" unless value.finite?
|
|
23
|
+
when Array
|
|
24
|
+
value.each_with_index { |item, i| check_value!(item, "#{path}[#{i}]") }
|
|
25
|
+
when Hash
|
|
26
|
+
value.each do |key, item|
|
|
27
|
+
unless key.is_a?(String)
|
|
28
|
+
raise InvalidArgumentsError,
|
|
29
|
+
"#{path} has the key #{key.inspect} (a #{key.class}); JSON only has String keys, so " \
|
|
30
|
+
"#perform would get back a different Hash -- use #{key.to_s.inspect}"
|
|
31
|
+
end
|
|
32
|
+
check_value!(item, "#{path}[#{key.inspect}]")
|
|
33
|
+
end
|
|
34
|
+
else
|
|
35
|
+
raise InvalidArgumentsError,
|
|
36
|
+
"#{path} is a #{value.class}, which doesn't survive a JSON round trip -- pass plain values " \
|
|
37
|
+
"(e.g. an id rather than a record, an ISO 8601 String rather than a Time)"
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
private_class_method :check_value!
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
module Monk
|
|
2
|
+
module Jobs
|
|
3
|
+
# Looking up a job class before Monk.freeze! (Monk.boot calls it) has
|
|
4
|
+
# sealed the registry -- from a worker Ractor the unfrozen registry
|
|
5
|
+
# couldn't even be read.
|
|
6
|
+
class NotFrozenError < StandardError
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
# A job_class name that isn't a Monk::Job subclass this process knows
|
|
10
|
+
# about: a job enqueued by a newer deploy, a renamed class, or a name
|
|
11
|
+
# that was never a job at all.
|
|
12
|
+
class UnknownJobError < StandardError
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
# Enqueueing (or claiming) before Monk::Jobs.configure has picked the
|
|
16
|
+
# database the queue lives in.
|
|
17
|
+
class NotConfiguredError < StandardError
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
# Monk::Jobs.clear! outside MONK_ENV=test: it would empty a real queue.
|
|
21
|
+
class ClearOutsideTestsError < StandardError
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# An ArgumentError, like Monk::Mail::InvalidMessageError: it's always
|
|
25
|
+
# the caller's input that's wrong.
|
|
26
|
+
class InvalidArgumentsError < ArgumentError
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
module Monk
|
|
2
|
+
# The class an app's jobs inherit from:
|
|
3
|
+
#
|
|
4
|
+
# class SendReceipt < Monk::Job
|
|
5
|
+
# queue "mailers" # optional; "default" otherwise
|
|
6
|
+
# priority 10 # optional; lower runs sooner, 0 otherwise
|
|
7
|
+
# max_attempts 3 # optional; 5 otherwise
|
|
8
|
+
# timeout 30 # optional; seconds a run may take, no limit otherwise
|
|
9
|
+
# never_retry KeyError # optional; errors that fail the job at once
|
|
10
|
+
#
|
|
11
|
+
# def self.perform(order_id)
|
|
12
|
+
# ...
|
|
13
|
+
# end
|
|
14
|
+
# end
|
|
15
|
+
#
|
|
16
|
+
# A class, not a block: a Class is always Ractor-shareable, so #perform
|
|
17
|
+
# is callable from every worker Ractor without the make_shareable rules
|
|
18
|
+
# a Proc has to meet (docs/design/ractor.md). Settings are inherited, so
|
|
19
|
+
# an app can put shared ones on its own base class.
|
|
20
|
+
class Job
|
|
21
|
+
DEFAULT_QUEUE = "default".freeze
|
|
22
|
+
DEFAULT_PRIORITY = 0
|
|
23
|
+
DEFAULT_MAX_ATTEMPTS = 5
|
|
24
|
+
|
|
25
|
+
# monk_jobs.priority and max_attempts are SMALLINT columns.
|
|
26
|
+
PRIORITY_RANGE = -32_768..32_767
|
|
27
|
+
MAX_ATTEMPTS_RANGE = 1..32_767
|
|
28
|
+
|
|
29
|
+
class << self
|
|
30
|
+
def inherited(subclass)
|
|
31
|
+
super
|
|
32
|
+
Monk::Jobs.record(subclass)
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def queue(name = nil)
|
|
36
|
+
return @queue || inherited_setting(:queue, DEFAULT_QUEUE) if name.nil?
|
|
37
|
+
|
|
38
|
+
unless name.is_a?(String) && !name.empty?
|
|
39
|
+
raise ArgumentError, "#{self.name || "a job"}'s queue must be a non-empty String, got #{name.inspect}"
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Frozen, so a worker Ractor can read it back off the class.
|
|
43
|
+
@queue = name.dup.freeze
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def priority(value = nil)
|
|
47
|
+
return @priority || inherited_setting(:priority, DEFAULT_PRIORITY) if value.nil?
|
|
48
|
+
|
|
49
|
+
@priority = checked_integer(:priority, value, PRIORITY_RANGE)
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def max_attempts(value = nil)
|
|
53
|
+
return @max_attempts || inherited_setting(:max_attempts, DEFAULT_MAX_ATTEMPTS) if value.nil?
|
|
54
|
+
|
|
55
|
+
@max_attempts = checked_integer(:max_attempts, value, MAX_ATTEMPTS_RANGE)
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Seconds one run of the job may take before the worker gives up on
|
|
59
|
+
# it: the run fails (and is retried like any other failure) and the
|
|
60
|
+
# worker resets its database connection, since an interrupted query
|
|
61
|
+
# keeps running on the server otherwise. nil, the default, means no
|
|
62
|
+
# limit.
|
|
63
|
+
def timeout(seconds = nil)
|
|
64
|
+
return @timeout || (equal?(Monk::Job) ? nil : superclass.timeout) if seconds.nil?
|
|
65
|
+
|
|
66
|
+
unless seconds.is_a?(Numeric) && seconds.positive?
|
|
67
|
+
raise ArgumentError,
|
|
68
|
+
"#{name || "a job"}'s timeout must be a positive number of seconds, got #{seconds.inspect}"
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
@timeout = seconds
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Errors that retrying can't fix: a job that raises one of these (or
|
|
75
|
+
# a subclass of one, matched the way rescue matches) fails at once
|
|
76
|
+
# instead of spending its remaining attempts. Adds to the parent
|
|
77
|
+
# class's list rather than replacing it, so an app's own base job
|
|
78
|
+
# class can declare errors for every job. With no arguments, returns
|
|
79
|
+
# the whole list -- frozen, so a worker Ractor can read it.
|
|
80
|
+
def never_retry(*error_classes)
|
|
81
|
+
inherited = equal?(Monk::Job) ? [] : superclass.never_retry
|
|
82
|
+
unless error_classes.empty?
|
|
83
|
+
error_classes.each do |error_class|
|
|
84
|
+
next if error_class.is_a?(Class) && error_class <= Exception
|
|
85
|
+
|
|
86
|
+
raise ArgumentError,
|
|
87
|
+
"#{name || "a job"}'s never_retry takes exception classes, got #{error_class.inspect}"
|
|
88
|
+
end
|
|
89
|
+
@never_retry = ((@never_retry || []) + error_classes).uniq.freeze
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
(inherited + (@never_retry || [])).uniq.freeze
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# SendReceipt.enqueue(order_id, wait: 60) -- see Monk::Jobs.enqueue.
|
|
96
|
+
def enqueue(*, **)
|
|
97
|
+
Monk::Jobs.enqueue(self, *, **)
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
def perform(*)
|
|
101
|
+
raise NotImplementedError, "#{name || "this job"} must define self.perform"
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
private
|
|
105
|
+
|
|
106
|
+
def inherited_setting(setting, default)
|
|
107
|
+
equal?(Monk::Job) ? default : superclass.public_send(setting)
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def checked_integer(setting, value, range)
|
|
111
|
+
return value if value.is_a?(Integer) && range.cover?(value)
|
|
112
|
+
|
|
113
|
+
raise ArgumentError, "#{name || "a job"}'s #{setting} must be an Integer in #{range}, got #{value.inspect}"
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
end
|