pgbus 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.bun-version +1 -0
- data/.claude/commands/architect.md +100 -0
- data/.claude/commands/github-review-comments.md +237 -0
- data/.claude/commands/lfg.md +271 -0
- data/.claude/commands/review-pr.md +69 -0
- data/.claude/commands/security.md +122 -0
- data/.claude/commands/tdd.md +148 -0
- data/.claude/rules/agents.md +49 -0
- data/.claude/rules/coding-style.md +91 -0
- data/.claude/rules/git-workflow.md +56 -0
- data/.claude/rules/performance.md +73 -0
- data/.claude/rules/testing.md +67 -0
- data/CHANGELOG.md +5 -0
- data/CLAUDE.md +80 -0
- data/CODE_OF_CONDUCT.md +10 -0
- data/LICENSE.txt +21 -0
- data/README.md +417 -0
- data/Rakefile +14 -0
- data/app/controllers/pgbus/api/stats_controller.rb +11 -0
- data/app/controllers/pgbus/application_controller.rb +35 -0
- data/app/controllers/pgbus/dashboard_controller.rb +27 -0
- data/app/controllers/pgbus/dead_letter_controller.rb +50 -0
- data/app/controllers/pgbus/events_controller.rb +23 -0
- data/app/controllers/pgbus/jobs_controller.rb +48 -0
- data/app/controllers/pgbus/processes_controller.rb +10 -0
- data/app/controllers/pgbus/queues_controller.rb +21 -0
- data/app/helpers/pgbus/application_helper.rb +69 -0
- data/app/views/layouts/pgbus/application.html.erb +76 -0
- data/app/views/pgbus/dashboard/_processes_table.html.erb +30 -0
- data/app/views/pgbus/dashboard/_queues_table.html.erb +39 -0
- data/app/views/pgbus/dashboard/_recent_failures.html.erb +33 -0
- data/app/views/pgbus/dashboard/_stats_cards.html.erb +28 -0
- data/app/views/pgbus/dashboard/show.html.erb +10 -0
- data/app/views/pgbus/dead_letter/_messages_table.html.erb +40 -0
- data/app/views/pgbus/dead_letter/index.html.erb +15 -0
- data/app/views/pgbus/dead_letter/show.html.erb +52 -0
- data/app/views/pgbus/events/index.html.erb +57 -0
- data/app/views/pgbus/events/show.html.erb +28 -0
- data/app/views/pgbus/jobs/_enqueued_table.html.erb +34 -0
- data/app/views/pgbus/jobs/_failed_table.html.erb +45 -0
- data/app/views/pgbus/jobs/index.html.erb +16 -0
- data/app/views/pgbus/jobs/show.html.erb +57 -0
- data/app/views/pgbus/processes/_processes_table.html.erb +37 -0
- data/app/views/pgbus/processes/index.html.erb +3 -0
- data/app/views/pgbus/queues/_queues_list.html.erb +41 -0
- data/app/views/pgbus/queues/index.html.erb +3 -0
- data/app/views/pgbus/queues/show.html.erb +49 -0
- data/bun.lock +18 -0
- data/config/routes.rb +45 -0
- data/docs/README.md +28 -0
- data/docs/switch_from_good_job.md +279 -0
- data/docs/switch_from_sidekiq.md +226 -0
- data/docs/switch_from_solid_queue.md +247 -0
- data/exe/pgbus +7 -0
- data/lib/generators/pgbus/install_generator.rb +56 -0
- data/lib/generators/pgbus/templates/migration.rb.erb +114 -0
- data/lib/generators/pgbus/templates/pgbus.yml.erb +74 -0
- data/lib/generators/pgbus/templates/pgbus_binstub.erb +7 -0
- data/lib/pgbus/active_job/adapter.rb +109 -0
- data/lib/pgbus/active_job/executor.rb +107 -0
- data/lib/pgbus/batch.rb +153 -0
- data/lib/pgbus/cli.rb +84 -0
- data/lib/pgbus/client.rb +162 -0
- data/lib/pgbus/concurrency/blocked_execution.rb +74 -0
- data/lib/pgbus/concurrency/semaphore.rb +66 -0
- data/lib/pgbus/concurrency.rb +65 -0
- data/lib/pgbus/config_loader.rb +27 -0
- data/lib/pgbus/configuration.rb +99 -0
- data/lib/pgbus/engine.rb +31 -0
- data/lib/pgbus/event.rb +31 -0
- data/lib/pgbus/event_bus/handler.rb +76 -0
- data/lib/pgbus/event_bus/publisher.rb +42 -0
- data/lib/pgbus/event_bus/registry.rb +54 -0
- data/lib/pgbus/event_bus/subscriber.rb +30 -0
- data/lib/pgbus/process/consumer.rb +113 -0
- data/lib/pgbus/process/dispatcher.rb +154 -0
- data/lib/pgbus/process/heartbeat.rb +71 -0
- data/lib/pgbus/process/signal_handler.rb +49 -0
- data/lib/pgbus/process/supervisor.rb +198 -0
- data/lib/pgbus/process/worker.rb +153 -0
- data/lib/pgbus/serializer.rb +43 -0
- data/lib/pgbus/version.rb +5 -0
- data/lib/pgbus/web/authentication.rb +24 -0
- data/lib/pgbus/web/data_source.rb +406 -0
- data/lib/pgbus.rb +49 -0
- data/package.json +9 -0
- data/sig/pgbus.rbs +4 -0
- metadata +198 -0
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
<div class="mb-6">
|
|
2
|
+
<%= link_to "← Back to Jobs".html_safe, pgbus.jobs_path, class: "text-sm text-indigo-600 hover:text-indigo-500" %>
|
|
3
|
+
</div>
|
|
4
|
+
|
|
5
|
+
<% if @job %>
|
|
6
|
+
<h1 class="text-2xl font-bold text-gray-900 mb-6">Failed Job #<%= @job["id"] %></h1>
|
|
7
|
+
|
|
8
|
+
<div class="rounded-lg bg-white shadow ring-1 ring-gray-200 p-6 mb-6">
|
|
9
|
+
<dl class="grid grid-cols-1 gap-4 sm:grid-cols-2">
|
|
10
|
+
<div>
|
|
11
|
+
<dt class="text-sm font-medium text-gray-500">Queue</dt>
|
|
12
|
+
<dd class="text-sm text-gray-900"><%= @job["queue_name"] %></dd>
|
|
13
|
+
</div>
|
|
14
|
+
<div>
|
|
15
|
+
<dt class="text-sm font-medium text-gray-500">Failed At</dt>
|
|
16
|
+
<dd class="text-sm text-gray-900"><%= @job["failed_at"] %></dd>
|
|
17
|
+
</div>
|
|
18
|
+
<div>
|
|
19
|
+
<dt class="text-sm font-medium text-gray-500">Error Class</dt>
|
|
20
|
+
<dd class="text-sm text-red-600 font-medium"><%= @job["error_class"] %></dd>
|
|
21
|
+
</div>
|
|
22
|
+
<div>
|
|
23
|
+
<dt class="text-sm font-medium text-gray-500">Retry Count</dt>
|
|
24
|
+
<dd class="text-sm text-gray-900"><%= @job["retry_count"] %></dd>
|
|
25
|
+
</div>
|
|
26
|
+
</dl>
|
|
27
|
+
</div>
|
|
28
|
+
|
|
29
|
+
<div class="rounded-lg bg-white shadow ring-1 ring-gray-200 p-6 mb-6">
|
|
30
|
+
<h2 class="text-sm font-medium text-gray-500 mb-2">Error Message</h2>
|
|
31
|
+
<p class="text-sm text-red-700"><%= @job["error_message"] %></p>
|
|
32
|
+
</div>
|
|
33
|
+
|
|
34
|
+
<% if @job["backtrace"].present? %>
|
|
35
|
+
<div class="rounded-lg bg-white shadow ring-1 ring-gray-200 p-6 mb-6">
|
|
36
|
+
<h2 class="text-sm font-medium text-gray-500 mb-2">Backtrace</h2>
|
|
37
|
+
<pre class="text-xs text-gray-600 bg-gray-50 rounded p-4 overflow-x-auto max-h-96"><%= @job["backtrace"] %></pre>
|
|
38
|
+
</div>
|
|
39
|
+
<% end %>
|
|
40
|
+
|
|
41
|
+
<div class="rounded-lg bg-white shadow ring-1 ring-gray-200 p-6 mb-6">
|
|
42
|
+
<h2 class="text-sm font-medium text-gray-500 mb-2">Payload</h2>
|
|
43
|
+
<pre class="text-xs text-gray-600 bg-gray-50 rounded p-4 overflow-x-auto"><%= JSON.pretty_generate(JSON.parse(@job["payload"])) rescue @job["payload"] %></pre>
|
|
44
|
+
</div>
|
|
45
|
+
|
|
46
|
+
<div class="flex space-x-2">
|
|
47
|
+
<%= button_to "Retry", pgbus.retry_job_path(@job["id"]), method: :post,
|
|
48
|
+
class: "rounded-md bg-indigo-600 px-4 py-2 text-sm font-medium text-white hover:bg-indigo-500" %>
|
|
49
|
+
<%= button_to "Discard", pgbus.discard_job_path(@job["id"]), method: :post,
|
|
50
|
+
class: "rounded-md bg-red-600 px-4 py-2 text-sm font-medium text-white hover:bg-red-500",
|
|
51
|
+
data: { turbo_confirm: "Discard this job permanently?" } %>
|
|
52
|
+
</div>
|
|
53
|
+
<% else %>
|
|
54
|
+
<div class="text-center py-12">
|
|
55
|
+
<p class="text-gray-400">Job not found</p>
|
|
56
|
+
</div>
|
|
57
|
+
<% end %>
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
<turbo-frame id="processes-list" data-auto-refresh src="<%= pgbus.processes_path(frame: 'list') %>">
|
|
2
|
+
<div class="overflow-hidden rounded-lg bg-white shadow ring-1 ring-gray-200">
|
|
3
|
+
<table class="min-w-full divide-y divide-gray-200">
|
|
4
|
+
<thead class="bg-gray-50">
|
|
5
|
+
<tr>
|
|
6
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Kind</th>
|
|
7
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Hostname</th>
|
|
8
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">PID</th>
|
|
9
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Status</th>
|
|
10
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Last Heartbeat</th>
|
|
11
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Metadata</th>
|
|
12
|
+
</tr>
|
|
13
|
+
</thead>
|
|
14
|
+
<tbody class="divide-y divide-gray-100">
|
|
15
|
+
<% @processes.each do |p| %>
|
|
16
|
+
<tr class="hover:bg-gray-50">
|
|
17
|
+
<td class="px-4 py-3 text-sm font-medium text-gray-900"><%= p[:kind] %></td>
|
|
18
|
+
<td class="px-4 py-3 text-sm text-gray-700"><%= p[:hostname] %></td>
|
|
19
|
+
<td class="px-4 py-3 text-sm font-mono text-gray-700"><%= p[:pid] %></td>
|
|
20
|
+
<td class="px-4 py-3 text-sm"><%= pgbus_status_badge(p[:healthy]) %></td>
|
|
21
|
+
<td class="px-4 py-3 text-sm text-gray-500"><%= pgbus_time_ago(p[:last_heartbeat_at]) %></td>
|
|
22
|
+
<td class="px-4 py-3 text-sm text-gray-500 font-mono text-xs max-w-xs truncate">
|
|
23
|
+
<% if p[:metadata].is_a?(Hash) %>
|
|
24
|
+
<% p[:metadata].each do |k, v| %>
|
|
25
|
+
<span class="inline-flex items-center rounded bg-gray-100 px-1.5 py-0.5 text-xs mr-1"><%= k %>: <%= v %></span>
|
|
26
|
+
<% end %>
|
|
27
|
+
<% end %>
|
|
28
|
+
</td>
|
|
29
|
+
</tr>
|
|
30
|
+
<% end %>
|
|
31
|
+
<% if @processes.empty? %>
|
|
32
|
+
<tr><td colspan="6" class="px-4 py-8 text-center text-sm text-gray-400">No processes running</td></tr>
|
|
33
|
+
<% end %>
|
|
34
|
+
</tbody>
|
|
35
|
+
</table>
|
|
36
|
+
</div>
|
|
37
|
+
</turbo-frame>
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
<turbo-frame id="queues-list" data-auto-refresh src="<%= pgbus.queues_path(frame: 'list') %>">
|
|
2
|
+
<div class="overflow-hidden rounded-lg bg-white shadow ring-1 ring-gray-200">
|
|
3
|
+
<table class="min-w-full divide-y divide-gray-200">
|
|
4
|
+
<thead class="bg-gray-50">
|
|
5
|
+
<tr>
|
|
6
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Queue</th>
|
|
7
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Depth</th>
|
|
8
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Visible</th>
|
|
9
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Oldest (s)</th>
|
|
10
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Newest (s)</th>
|
|
11
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Total Ever</th>
|
|
12
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500">Actions</th>
|
|
13
|
+
</tr>
|
|
14
|
+
</thead>
|
|
15
|
+
<tbody class="divide-y divide-gray-100">
|
|
16
|
+
<% @queues.each do |q| %>
|
|
17
|
+
<tr class="hover:bg-gray-50">
|
|
18
|
+
<td class="px-4 py-3 text-sm">
|
|
19
|
+
<%= link_to q[:name], pgbus.queue_path(name: q[:name]), class: "font-medium text-indigo-600 hover:text-indigo-500", data: { turbo_frame: "_top" } %>
|
|
20
|
+
<%= pgbus_queue_badge(q[:name]) %>
|
|
21
|
+
</td>
|
|
22
|
+
<td class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
|
|
23
|
+
<td class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
|
|
24
|
+
<td class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_msg_age_sec] || "—" %></td>
|
|
25
|
+
<td class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:newest_msg_age_sec] || "—" %></td>
|
|
26
|
+
<td class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
|
|
27
|
+
<td class="px-4 py-3 text-sm text-right">
|
|
28
|
+
<%= button_to "Purge", pgbus.purge_queue_path(name: q[:name]),
|
|
29
|
+
method: :post,
|
|
30
|
+
class: "text-xs text-red-600 hover:text-red-800 font-medium",
|
|
31
|
+
data: { turbo_confirm: "Purge all messages from #{q[:name]}?", turbo_frame: "_top" } %>
|
|
32
|
+
</td>
|
|
33
|
+
</tr>
|
|
34
|
+
<% end %>
|
|
35
|
+
<% if @queues.empty? %>
|
|
36
|
+
<tr><td colspan="7" class="px-4 py-8 text-center text-sm text-gray-400">No queues found</td></tr>
|
|
37
|
+
<% end %>
|
|
38
|
+
</tbody>
|
|
39
|
+
</table>
|
|
40
|
+
</div>
|
|
41
|
+
</turbo-frame>
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
<div class="flex items-center justify-between mb-6">
|
|
2
|
+
<div>
|
|
3
|
+
<h1 class="text-2xl font-bold text-gray-900"><%= @queue&.dig(:name) || params[:name] %></h1>
|
|
4
|
+
<% if @queue %>
|
|
5
|
+
<p class="text-sm text-gray-500 mt-1">
|
|
6
|
+
Depth: <span class="font-mono"><%= @queue[:queue_length] %></span> |
|
|
7
|
+
Visible: <span class="font-mono"><%= @queue[:queue_visible_length] %></span> |
|
|
8
|
+
Total: <span class="font-mono"><%= pgbus_number(@queue[:total_messages]) %></span>
|
|
9
|
+
</p>
|
|
10
|
+
<% end %>
|
|
11
|
+
</div>
|
|
12
|
+
<div class="flex space-x-2">
|
|
13
|
+
<%= button_to "Purge Queue", pgbus.purge_queue_path(name: params[:name]),
|
|
14
|
+
method: :post,
|
|
15
|
+
class: "rounded-md bg-red-600 px-3 py-2 text-sm font-medium text-white hover:bg-red-500",
|
|
16
|
+
data: { turbo_confirm: "Purge all messages?" } %>
|
|
17
|
+
</div>
|
|
18
|
+
</div>
|
|
19
|
+
|
|
20
|
+
<!-- Messages in Queue -->
|
|
21
|
+
<div class="overflow-hidden rounded-lg bg-white shadow ring-1 ring-gray-200">
|
|
22
|
+
<table class="min-w-full divide-y divide-gray-200">
|
|
23
|
+
<thead class="bg-gray-50">
|
|
24
|
+
<tr>
|
|
25
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">ID</th>
|
|
26
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Enqueued</th>
|
|
27
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Reads</th>
|
|
28
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">VT</th>
|
|
29
|
+
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500">Payload</th>
|
|
30
|
+
</tr>
|
|
31
|
+
</thead>
|
|
32
|
+
<tbody class="divide-y divide-gray-100">
|
|
33
|
+
<% @messages.each do |m| %>
|
|
34
|
+
<tr class="hover:bg-gray-50">
|
|
35
|
+
<td class="px-4 py-3 text-sm font-mono text-gray-900"><%= m[:msg_id] %></td>
|
|
36
|
+
<td class="px-4 py-3 text-sm text-gray-500"><%= pgbus_time_ago(m[:enqueued_at]) %></td>
|
|
37
|
+
<td class="px-4 py-3 text-sm text-gray-500"><%= m[:read_ct] %></td>
|
|
38
|
+
<td class="px-4 py-3 text-sm text-gray-500"><%= pgbus_time_ago(m[:vt]) %></td>
|
|
39
|
+
<td class="px-4 py-3 text-sm text-gray-600 font-mono text-xs max-w-md truncate">
|
|
40
|
+
<%= pgbus_json_preview(m[:message]) %>
|
|
41
|
+
</td>
|
|
42
|
+
</tr>
|
|
43
|
+
<% end %>
|
|
44
|
+
<% if @messages.empty? %>
|
|
45
|
+
<tr><td colspan="5" class="px-4 py-8 text-center text-sm text-gray-400">Queue is empty</td></tr>
|
|
46
|
+
<% end %>
|
|
47
|
+
</tbody>
|
|
48
|
+
</table>
|
|
49
|
+
</div>
|
data/bun.lock
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
{
|
|
2
|
+
"lockfileVersion": 1,
|
|
3
|
+
"configVersion": 0,
|
|
4
|
+
"workspaces": {
|
|
5
|
+
"": {
|
|
6
|
+
"devDependencies": {
|
|
7
|
+
"playwright": "^1.50.0",
|
|
8
|
+
},
|
|
9
|
+
},
|
|
10
|
+
},
|
|
11
|
+
"packages": {
|
|
12
|
+
"fsevents": ["fsevents@2.3.2", "", { "os": "darwin" }, "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="],
|
|
13
|
+
|
|
14
|
+
"playwright": ["playwright@1.58.2", "", { "dependencies": { "playwright-core": "1.58.2" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": "cli.js" }, "sha512-vA30H8Nvkq/cPBnNw4Q8TWz1EJyqgpuinBcHET0YVJVFldr8JDNiU9LaWAE1KqSkRYazuaBhTpB5ZzShOezQ6A=="],
|
|
15
|
+
|
|
16
|
+
"playwright-core": ["playwright-core@1.58.2", "", { "bin": "cli.js" }, "sha512-yZkEtftgwS8CsfYo7nm0KE8jsvm6i/PTgVtB8DL726wNf6H2IMsDuxCpJj59KDaxCtSnrWan2AeDqM7JBaultg=="],
|
|
17
|
+
}
|
|
18
|
+
}
|
data/config/routes.rb
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
Pgbus::Engine.routes.draw do
|
|
4
|
+
root to: "dashboard#show"
|
|
5
|
+
|
|
6
|
+
resources :queues, only: %i[index show], param: :name do
|
|
7
|
+
member do
|
|
8
|
+
post :purge
|
|
9
|
+
end
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
resources :jobs, only: %i[index show] do
|
|
13
|
+
member do
|
|
14
|
+
post :retry
|
|
15
|
+
post :discard
|
|
16
|
+
end
|
|
17
|
+
collection do
|
|
18
|
+
post :retry_all
|
|
19
|
+
post :discard_all
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
resources :processes, only: [:index]
|
|
24
|
+
|
|
25
|
+
resources :events, only: %i[index show] do
|
|
26
|
+
member do
|
|
27
|
+
post :replay
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
resources :dead_letter, only: %i[index show], path: "dlq" do
|
|
32
|
+
member do
|
|
33
|
+
post :retry
|
|
34
|
+
post :discard
|
|
35
|
+
end
|
|
36
|
+
collection do
|
|
37
|
+
post :retry_all
|
|
38
|
+
post :discard_all
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
namespace :api do
|
|
43
|
+
get :stats, to: "stats#show"
|
|
44
|
+
end
|
|
45
|
+
end
|
data/docs/README.md
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Pgbus Documentation
|
|
2
|
+
|
|
3
|
+
## Migration guides
|
|
4
|
+
|
|
5
|
+
Switching to Pgbus from another job backend? These guides cover what changes, what stays the same, and what to watch out for.
|
|
6
|
+
|
|
7
|
+
| Guide | Backend | Key differences |
|
|
8
|
+
|-------|---------|-----------------|
|
|
9
|
+
| [Switch from Sidekiq](switch_from_sidekiq.md) | Sidekiq (+ Pro/Enterprise) | Remove Redis, convert native workers to ActiveJob, replace middleware with callbacks |
|
|
10
|
+
| [Switch from SolidQueue](switch_from_solid_queue.md) | SolidQueue | Similar architecture (PostgreSQL + `SKIP LOCKED`), swap config format, gain LISTEN/NOTIFY + worker recycling |
|
|
11
|
+
| [Switch from GoodJob](switch_from_good_job.md) | GoodJob | Both PostgreSQL-native with LISTEN/NOTIFY, swap advisory locks for PGMQ visibility timeouts, gain worker recycling |
|
|
12
|
+
|
|
13
|
+
## Feature comparison
|
|
14
|
+
|
|
15
|
+
| Feature | Sidekiq | SolidQueue | GoodJob | Pgbus |
|
|
16
|
+
|---------|---------|------------|---------|-------|
|
|
17
|
+
| Infrastructure | Redis | PostgreSQL | PostgreSQL | PostgreSQL (PGMQ) |
|
|
18
|
+
| ActiveJob adapter | Yes | Yes | Yes | Yes |
|
|
19
|
+
| Bulk enqueue | No | Yes | Yes | Yes |
|
|
20
|
+
| LISTEN/NOTIFY | N/A | No (polling only) | Yes | Yes |
|
|
21
|
+
| Dead letter queues | No (retries only) | No | No | Yes |
|
|
22
|
+
| Worker recycling | No | No | No | Yes |
|
|
23
|
+
| Event bus | No | No | No | Yes |
|
|
24
|
+
| Idempotent events | No | No | No | Yes |
|
|
25
|
+
| Concurrency controls | Enterprise | `limits_concurrency` | `good_job_control_concurrency_with` | `Pgbus::Concurrency` |
|
|
26
|
+
| Recurring/cron jobs | `sidekiq-cron` gem | `config/recurring.yml` | `config.good_job.cron` | Planned |
|
|
27
|
+
| Batches | Pro | No | `GoodJob::Batch` | `Pgbus::Batch` |
|
|
28
|
+
| Web dashboard | `Sidekiq::Web` | Mission Control | `GoodJob::Engine` | `Pgbus::Engine` |
|
|
@@ -0,0 +1,279 @@
|
|
|
1
|
+
# Switch from GoodJob to Pgbus
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
GoodJob and Pgbus are both PostgreSQL-native job processors with LISTEN/NOTIFY support. The biggest architectural difference: GoodJob uses advisory locks and a `good_jobs` table, while Pgbus uses PGMQ (a dedicated message queue extension) with visibility timeouts. Both are pure ActiveJob adapters.
|
|
6
|
+
|
|
7
|
+
**Effort estimate:** Low if you use standard ActiveJob. Medium if you rely on GoodJob's concurrency controls, batches, or cron.
|
|
8
|
+
|
|
9
|
+
## Step 1: Update dependencies
|
|
10
|
+
|
|
11
|
+
```ruby
|
|
12
|
+
# Gemfile
|
|
13
|
+
|
|
14
|
+
# Remove
|
|
15
|
+
gem "good_job"
|
|
16
|
+
|
|
17
|
+
# Add
|
|
18
|
+
gem "pgbus"
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
bundle install
|
|
23
|
+
rails generate pgbus:install
|
|
24
|
+
rails db:migrate
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## Step 2: Switch the adapter
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
# config/application.rb
|
|
31
|
+
|
|
32
|
+
# Before
|
|
33
|
+
config.active_job.queue_adapter = :good_job
|
|
34
|
+
|
|
35
|
+
# After
|
|
36
|
+
config.active_job.queue_adapter = :pgbus
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Remove GoodJob configuration:
|
|
40
|
+
|
|
41
|
+
```ruby
|
|
42
|
+
# Before: Remove this block
|
|
43
|
+
config.good_job = {
|
|
44
|
+
execution_mode: :async,
|
|
45
|
+
queues: "critical:5;default:3;*:1",
|
|
46
|
+
max_threads: 5,
|
|
47
|
+
poll_interval: 10,
|
|
48
|
+
enable_cron: true,
|
|
49
|
+
enable_listen_notify: true,
|
|
50
|
+
# ...
|
|
51
|
+
}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Step 3: Convert worker configuration
|
|
55
|
+
|
|
56
|
+
```ruby
|
|
57
|
+
# Before: GoodJob config (in application.rb or environment files)
|
|
58
|
+
config.good_job = {
|
|
59
|
+
execution_mode: :external, # or :async
|
|
60
|
+
queues: "critical:5;default:3;low:1",
|
|
61
|
+
poll_interval: 10,
|
|
62
|
+
max_threads: 5,
|
|
63
|
+
shutdown_timeout: 30,
|
|
64
|
+
}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
```yaml
|
|
68
|
+
# After: config/pgbus.yml
|
|
69
|
+
production:
|
|
70
|
+
workers:
|
|
71
|
+
- queues: [critical]
|
|
72
|
+
threads: 5
|
|
73
|
+
- queues: [default]
|
|
74
|
+
threads: 3
|
|
75
|
+
- queues: [low]
|
|
76
|
+
threads: 1
|
|
77
|
+
max_retries: 5
|
|
78
|
+
max_jobs_per_worker: 10000
|
|
79
|
+
max_memory_mb: 512
|
|
80
|
+
max_worker_lifetime: 3600
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### Configuration mapping
|
|
84
|
+
|
|
85
|
+
| GoodJob | Pgbus | Notes |
|
|
86
|
+
|---------|-------|-------|
|
|
87
|
+
| `execution_mode: :external` | `bundle exec pgbus start` | Separate process (recommended for production) |
|
|
88
|
+
| `execution_mode: :async` | N/A | Pgbus always runs as separate processes |
|
|
89
|
+
| `queues: "critical:5;default:3"` | `workers:` array | One entry per worker process |
|
|
90
|
+
| `max_threads` | `threads` per worker | Per-worker, not global |
|
|
91
|
+
| `poll_interval` | `polling_interval` | Pgbus defaults to 0.1s; LISTEN/NOTIFY is primary |
|
|
92
|
+
| `enable_listen_notify` | `listen_notify` | Both support LISTEN/NOTIFY |
|
|
93
|
+
| `shutdown_timeout` | Handled by supervisor | Graceful shutdown on SIGTERM |
|
|
94
|
+
| `on_thread_error` | Configure via `Pgbus.logger` | Error reporting via logging |
|
|
95
|
+
| `preserve_job_records` | Always archived | PGMQ archives completed messages |
|
|
96
|
+
|
|
97
|
+
## Step 4: Remove concurrency controls
|
|
98
|
+
|
|
99
|
+
GoodJob's concurrency extensions are GoodJob-specific. Remove them:
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
# Before
|
|
103
|
+
class ProcessOrderJob < ApplicationJob
|
|
104
|
+
include GoodJob::ActiveJobExtensions::Concurrency
|
|
105
|
+
|
|
106
|
+
good_job_control_concurrency_with(
|
|
107
|
+
total_limit: 1,
|
|
108
|
+
enqueue_limit: 2,
|
|
109
|
+
perform_limit: 1,
|
|
110
|
+
enqueue_throttle: [10, 1.minute],
|
|
111
|
+
perform_throttle: [100, 1.hour],
|
|
112
|
+
key: -> { "ProcessOrder-#{arguments.first.id}" }
|
|
113
|
+
)
|
|
114
|
+
|
|
115
|
+
def perform(order)
|
|
116
|
+
# ...
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
# After
|
|
123
|
+
class ProcessOrderJob < ApplicationJob
|
|
124
|
+
def perform(order)
|
|
125
|
+
# ...
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
> Pgbus supports concurrency controls via `Pgbus::Concurrency`:
|
|
131
|
+
> ```ruby
|
|
132
|
+
> class ProcessOrderJob < ApplicationJob
|
|
133
|
+
> include Pgbus::Concurrency
|
|
134
|
+
> limits_concurrency to: 1,
|
|
135
|
+
> key: -> { "ProcessOrder-#{arguments.first.id}" },
|
|
136
|
+
> duration: 15.minutes,
|
|
137
|
+
> on_conflict: :block
|
|
138
|
+
> def perform(order)
|
|
139
|
+
> # ...
|
|
140
|
+
> end
|
|
141
|
+
> end
|
|
142
|
+
> ```
|
|
143
|
+
|
|
144
|
+
## Step 5: Migrate batches
|
|
145
|
+
|
|
146
|
+
If you use `GoodJob::Batch`:
|
|
147
|
+
|
|
148
|
+
```ruby
|
|
149
|
+
# Before: GoodJob batches
|
|
150
|
+
GoodJob::Batch.enqueue(
|
|
151
|
+
on_finish: BatchCallbackJob,
|
|
152
|
+
on_success: SuccessNotifyJob,
|
|
153
|
+
description: "Import users"
|
|
154
|
+
) do
|
|
155
|
+
users.each { |u| ImportUserJob.perform_later(u) }
|
|
156
|
+
end
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Pgbus does not yet have batch support. Workaround using a coordinator job:
|
|
160
|
+
|
|
161
|
+
```ruby
|
|
162
|
+
# After: Coordinator pattern
|
|
163
|
+
class ImportUsersJob < ApplicationJob
|
|
164
|
+
def perform(user_ids)
|
|
165
|
+
user_ids.each { |id| ImportUserJob.perform_later(id) }
|
|
166
|
+
# Track completion via a counter in the database or Redis
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## Step 6: Migrate cron / recurring jobs
|
|
172
|
+
|
|
173
|
+
```ruby
|
|
174
|
+
# Before: GoodJob cron
|
|
175
|
+
config.good_job.enable_cron = true
|
|
176
|
+
config.good_job.cron = {
|
|
177
|
+
daily_cleanup: {
|
|
178
|
+
cron: "0 2 * * *",
|
|
179
|
+
class: "CleanupJob",
|
|
180
|
+
set: { priority: -10, queue: "maintenance" },
|
|
181
|
+
},
|
|
182
|
+
hourly_sync: {
|
|
183
|
+
cron: "0 * * * *",
|
|
184
|
+
class: "SyncJob",
|
|
185
|
+
args: [42],
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Pgbus does not yet have built-in recurring task support. Options:
|
|
191
|
+
|
|
192
|
+
1. **Use the `whenever` gem**:
|
|
193
|
+
```ruby
|
|
194
|
+
# config/schedule.rb
|
|
195
|
+
every 1.day, at: "2:00 am" do
|
|
196
|
+
runner "CleanupJob.perform_later"
|
|
197
|
+
end
|
|
198
|
+
every :hour do
|
|
199
|
+
runner "SyncJob.perform_later(42)"
|
|
200
|
+
end
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
2. **Use system cron** directly:
|
|
204
|
+
```cron
|
|
205
|
+
0 2 * * * cd /app && bin/rails runner "CleanupJob.perform_later"
|
|
206
|
+
0 * * * * cd /app && bin/rails runner "SyncJob.perform_later(42)"
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
3. Wait for Pgbus recurring task support (planned).
|
|
210
|
+
|
|
211
|
+
## Step 7: Replace the dashboard
|
|
212
|
+
|
|
213
|
+
```ruby
|
|
214
|
+
# Before: config/routes.rb
|
|
215
|
+
mount GoodJob::Engine => "good_job"
|
|
216
|
+
|
|
217
|
+
# After:
|
|
218
|
+
mount Pgbus::Engine => "/pgbus"
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## Step 8: Update process management
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
# Before (external mode)
|
|
225
|
+
bundle exec good_job start
|
|
226
|
+
|
|
227
|
+
# After
|
|
228
|
+
bundle exec pgbus start
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
If you ran GoodJob in `async` mode (in-process), note that Pgbus always runs as separate forked processes managed by a supervisor. Update your deployment accordingly -- you need `bundle exec pgbus start` as a separate process.
|
|
232
|
+
|
|
233
|
+
## Step 9: Clean up GoodJob tables
|
|
234
|
+
|
|
235
|
+
After verifying Pgbus is processing correctly and GoodJob's tables are drained:
|
|
236
|
+
|
|
237
|
+
```ruby
|
|
238
|
+
class RemoveGoodJob < ActiveRecord::Migration[7.1]
|
|
239
|
+
def up
|
|
240
|
+
drop_table :good_jobs, if_exists: true
|
|
241
|
+
drop_table :good_job_batches, if_exists: true
|
|
242
|
+
drop_table :good_job_executions, if_exists: true
|
|
243
|
+
drop_table :good_job_processes, if_exists: true
|
|
244
|
+
drop_table :good_job_settings, if_exists: true
|
|
245
|
+
end
|
|
246
|
+
end
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
## What you gain
|
|
250
|
+
|
|
251
|
+
- **Dead letter queues** -- GoodJob retries in-place and marks jobs as discarded. Pgbus routes failures to dedicated `_dlq` queues for clear operational visibility.
|
|
252
|
+
- **Worker recycling** -- GoodJob workers run indefinitely. Pgbus recycles by job count, memory, or lifetime to prevent memory bloat.
|
|
253
|
+
- **Event bus** -- AMQP-style pub/sub with topic routing and idempotent handlers.
|
|
254
|
+
- **PGMQ** -- purpose-built message queue extension with atomic read/archive/delete, visibility timeouts, and `SKIP LOCKED` under the hood.
|
|
255
|
+
- **Supervisor/fork model** -- isolated worker processes. A memory leak or crash in one worker doesn't affect others.
|
|
256
|
+
|
|
257
|
+
## What you lose (for now)
|
|
258
|
+
|
|
259
|
+
| GoodJob feature | Status in Pgbus |
|
|
260
|
+
|-----------------|-----------------|
|
|
261
|
+
| Concurrency controls (`good_job_control_concurrency_with`) | `Pgbus::Concurrency` with `limits_concurrency` DSL |
|
|
262
|
+
| Throttling (`enqueue_throttle`, `perform_throttle`) | Planned |
|
|
263
|
+
| Batches (`GoodJob::Batch`) | `Pgbus::Batch` with on_finish/on_success/on_discard callbacks |
|
|
264
|
+
| Cron / recurring jobs | Planned |
|
|
265
|
+
| Async execution mode (in-process) | Not planned (forked processes only) |
|
|
266
|
+
| Capsules (isolated thread pools) | Workers are isolated by design (forked processes) |
|
|
267
|
+
| Advisory locks | Replaced by PGMQ visibility timeouts |
|
|
268
|
+
|
|
269
|
+
## Gotchas
|
|
270
|
+
|
|
271
|
+
1. **PgBouncer**: Both GoodJob and Pgbus use LISTEN/NOTIFY, which requires session-mode PgBouncer or direct connections. If you already had GoodJob working with PgBouncer, the same configuration applies.
|
|
272
|
+
|
|
273
|
+
2. **No async mode**: GoodJob can run in-process (`:async` mode) alongside your Rails app server. Pgbus requires a separate supervisor process. Make sure your deployment runs `bundle exec pgbus start` alongside your web server.
|
|
274
|
+
|
|
275
|
+
3. **Advisory locks vs. visibility timeouts**: GoodJob uses PostgreSQL advisory locks to claim jobs. Pgbus uses PGMQ's visibility timeout -- a claimed message becomes invisible for `visibility_timeout` seconds. If a worker crashes without archiving the message, it automatically becomes available again after the timeout expires. This is more resilient than advisory locks, which release on disconnect.
|
|
276
|
+
|
|
277
|
+
4. **Job record preservation**: GoodJob has `preserve_job_records` for keeping completed job records. PGMQ automatically archives completed messages to `a_pgbus_*` tables, which serve a similar purpose for debugging and auditing.
|
|
278
|
+
|
|
279
|
+
5. **Queue naming**: GoodJob uses bare queue names. Pgbus prefixes all queues (`pgbus_default`). Your `queue_as` declarations work unchanged -- the prefix is applied automatically.
|