sidekiq_publisher 1.6.4 → 1.8.0.rc0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/CODEOWNERS +1 -0
- data/Appraisals +18 -0
- data/CHANGELOG.md +12 -0
- data/Dockerfile +4 -0
- data/README.md +43 -0
- data/docker-compose.yml +43 -0
- data/gemfiles/rails_6.1_sidekiq_5.2.gemfile +9 -0
- data/gemfiles/rails_6.1_sidekiq_6.0.gemfile +9 -0
- data/gemfiles/rails_6.1_sidekiq_6.1.gemfile +9 -0
- data/lib/sidekiq_publisher.rb +4 -0
- data/lib/sidekiq_publisher/datadog_apm.rb +106 -0
- data/lib/sidekiq_publisher/exception_reporter.rb +15 -0
- data/lib/sidekiq_publisher/instrumenter.rb +13 -0
- data/lib/sidekiq_publisher/job.rb +4 -3
- data/lib/sidekiq_publisher/metrics_reporter.rb +40 -0
- data/lib/sidekiq_publisher/publisher.rb +25 -24
- data/lib/sidekiq_publisher/report_unpublished_count.rb +15 -4
- data/lib/sidekiq_publisher/runner.rb +23 -14
- data/lib/sidekiq_publisher/version.rb +1 -1
- data/sidekiq_publisher.gemspec +2 -1
- metadata +32 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5e164c321e567b1b58344c77cfb0dbb9889e1fbc6ec7440a02c3c6460290d73a
|
4
|
+
data.tar.gz: f7c13b3cc7e994df855b0de0aeeab9a34b68accd1a4ae1d50485f907ccfdd05c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 0eb8e48640b4afc46d9f5e55c77e691dd391a6fbdace88f5704acc899ee82699f6dfbe5038ed426a46bb4bbf73bd377cd6b89a1e58c148961347bdb61f3bcf80
|
7
|
+
data.tar.gz: bc3fbdacdc63d7217ed17222acdb4b8e67cb02f21d15c931e16c866ba3ae46913c6afbc777b1c7f09a4c8bc49e1314e854ec55e3014fd9fc71943bdd3ba05b20
|
data/.github/CODEOWNERS
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
* @ezcater/core-platform @tjwp
|
data/Appraisals
CHANGED
@@ -52,3 +52,21 @@ appraise "rails-6.0-sidekiq-6.1" do
|
|
52
52
|
gem "activesupport", "~> 6.0.0"
|
53
53
|
gem "sidekiq", "~> 6.1.0"
|
54
54
|
end
|
55
|
+
|
56
|
+
appraise "rails-6.1-sidekiq-5.2" do
|
57
|
+
gem "activejob", "~> 6.1.0"
|
58
|
+
gem "activesupport", "~> 6.1.0"
|
59
|
+
gem "sidekiq", "~> 5.2.0"
|
60
|
+
end
|
61
|
+
|
62
|
+
appraise "rails-6.1-sidekiq-6.0" do
|
63
|
+
gem "activejob", "~> 6.1.0"
|
64
|
+
gem "activesupport", "~> 6.1.0"
|
65
|
+
gem "sidekiq", "~> 6.0.1"
|
66
|
+
end
|
67
|
+
|
68
|
+
appraise "rails-6.1-sidekiq-6.1" do
|
69
|
+
gem "activejob", "~> 6.1.0"
|
70
|
+
gem "activesupport", "~> 6.1.0"
|
71
|
+
gem "sidekiq", "~> 6.1.0"
|
72
|
+
end
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,17 @@
|
|
1
1
|
# sidekiq_publisher
|
2
2
|
|
3
|
+
## v1.8.0 (unreleased)
|
4
|
+
- Extend support to Rails 6.1.
|
5
|
+
|
6
|
+
## v1.7.1
|
7
|
+
- Gracefully handle database connection errors in ReportUnpublishedCount by attempting to reconnect.
|
8
|
+
|
9
|
+
## v1.7.0
|
10
|
+
- Add instrumentation using `ActiveSupport::Notifications`.
|
11
|
+
- Reimplement `metrics_reporter` and `exception_reporter` support using
|
12
|
+
`ActiveSupport::Subscriber`.
|
13
|
+
- Add optional integration with Datadog APM.
|
14
|
+
|
3
15
|
## v1.6.4
|
4
16
|
- Expand sidekiq support to v5.0.x-v6.x.x.
|
5
17
|
|
data/Dockerfile
ADDED
data/README.md
CHANGED
@@ -72,6 +72,34 @@ SidekiqPublisher::ReportUnpublishedCount.call
|
|
72
72
|
It is recommended to call this method periodically using something like
|
73
73
|
cron or [clockwork](https://github.com/Rykian/clockwork).
|
74
74
|
|
75
|
+
## Instrumentation
|
76
|
+
|
77
|
+
Instrumentation of this library is implemented using
|
78
|
+
[ActiveSupport::Notifications](https://api.rubyonrails.org/classes/ActiveSupport/Notifications.html).
|
79
|
+
|
80
|
+
The support for the configurable [metrics_reporter](lib/sidekiq_publisher/metrics_reporter.rb) and
|
81
|
+
[exception_reporter](lib/sidekiq_publisher/exception_reporter.rb) options is implemented using
|
82
|
+
[ActiveSupport::Subscriber](https://api.rubyonrails.org/classes/ActiveSupport/Subscriber.html).
|
83
|
+
|
84
|
+
If an alternate integration is required for metrics or error reporting then it can be implemented using outside this
|
85
|
+
library based on these examples.
|
86
|
+
|
87
|
+
### Tracing
|
88
|
+
|
89
|
+
The instrumentation in the library also supports integration with application tracing products, such as
|
90
|
+
[Datadog APM](https://www.datadoghq.com/product/apm/).
|
91
|
+
|
92
|
+
There is an optional integration with Datadog APM that can be required:
|
93
|
+
|
94
|
+
```ruby
|
95
|
+
require "sidekiq_publisher/datadog_apm"
|
96
|
+
```
|
97
|
+
|
98
|
+
This file must be required in addition including the `sidekiq_publisher` gem or requiring `sidekiq_publisher`.
|
99
|
+
|
100
|
+
This integration covers all of the sections of the library that are instrumented and serves an
|
101
|
+
[example](lib/sidekiq_publisher/datadog_apm) for implementing trace reporting for other products outside this library.
|
102
|
+
|
75
103
|
## Usage
|
76
104
|
|
77
105
|
### ActiveJob Adapter
|
@@ -108,6 +136,21 @@ class MyJob < ApplicationJob
|
|
108
136
|
end
|
109
137
|
```
|
110
138
|
|
139
|
+
#### ActiveJob Exception Reporting
|
140
|
+
|
141
|
+
Many exception monitoring service (e.g. Sentry, Airbrake, Honeybadger, etc) already provide basic integration support for `Sidekiq`.
|
142
|
+
These integration should also work with `SidekiqPublisher`.
|
143
|
+
However, you may need to explicitly include
|
144
|
+
`ActiveJob::QueueAdapters::SidekiqPublisherAdapter` as a compatible adapter for this to work properly.
|
145
|
+
|
146
|
+
Alternatively, you can manually report the exception:
|
147
|
+
|
148
|
+
```ruby
|
149
|
+
retry_on SomeError, attempts: 10 do |_job, exception|
|
150
|
+
Raven.capture_exception(exception, extra: { custom: :foo }) # Reporting using the Sentry gem
|
151
|
+
end
|
152
|
+
```
|
153
|
+
|
111
154
|
### SidekiqPublisher::Worker
|
112
155
|
|
113
156
|
Sidekiq workers are usually defined by including `Sidekiq::Worker` in a class.
|
data/docker-compose.yml
ADDED
@@ -0,0 +1,43 @@
|
|
1
|
+
version: "3.4"
|
2
|
+
volumes:
|
3
|
+
bundle-volume:
|
4
|
+
shared-volume:
|
5
|
+
x-environment: &default-environment
|
6
|
+
PRYRC: /usr/src/app/.docker-pryrc
|
7
|
+
BUNDLE_IGNORE_CONFIG: 1
|
8
|
+
BUNDLE_DISABLE_SHARED_GEMS: "true"
|
9
|
+
PGUSER: postgres
|
10
|
+
PGHOST: sidekiq-publisher-postgres
|
11
|
+
PGPORT: 5432
|
12
|
+
PGDATABASE: sidekiq_publisher_test
|
13
|
+
REDIS_URL: redis://sidekiq-publisher-redis:6379
|
14
|
+
x-service: &default-service
|
15
|
+
build:
|
16
|
+
context: .
|
17
|
+
args:
|
18
|
+
- BUNDLE_EZCATER__JFROG__IO
|
19
|
+
volumes:
|
20
|
+
- .:/usr/src/gem
|
21
|
+
- bundle-volume:/usr/local/bundle:delegated
|
22
|
+
- shared-volume:/usr/src/shared:delegated
|
23
|
+
tty: true
|
24
|
+
stdin_open: true
|
25
|
+
services:
|
26
|
+
sidekiq-publisher-redis:
|
27
|
+
container_name: sidekiq-publisher-redis_1
|
28
|
+
image: redis:5.0.9-alpine
|
29
|
+
sidekiq-publisher-postgres:
|
30
|
+
container_name: sidekiq-publisher-postgres_1
|
31
|
+
image: postgres:11.6
|
32
|
+
environment:
|
33
|
+
POSTGRES_USER: postgres
|
34
|
+
POSTGRES_DB: sidekiq_publisher_test
|
35
|
+
sidekiq-publisher-console:
|
36
|
+
<<: *default-service
|
37
|
+
container_name: sidekiq-publisher-console_1
|
38
|
+
environment:
|
39
|
+
<<: *default-environment
|
40
|
+
command: bash
|
41
|
+
depends_on:
|
42
|
+
- sidekiq-publisher-redis
|
43
|
+
- sidekiq-publisher-postgres
|
data/lib/sidekiq_publisher.rb
CHANGED
@@ -1,6 +1,10 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "active_support"
|
3
4
|
require "sidekiq_publisher/version"
|
5
|
+
require "sidekiq_publisher/instrumenter"
|
6
|
+
require "sidekiq_publisher/metrics_reporter"
|
7
|
+
require "sidekiq_publisher/exception_reporter"
|
4
8
|
require "sidekiq_publisher/report_unpublished_count"
|
5
9
|
require "sidekiq_publisher/job"
|
6
10
|
require "sidekiq_publisher/worker"
|
@@ -0,0 +1,106 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "active_support/subscriber"
|
4
|
+
require "ddtrace"
|
5
|
+
|
6
|
+
module SidekiqPublisher
|
7
|
+
module DatadogAPM
|
8
|
+
OPERATION = "sidekiq_publisher"
|
9
|
+
|
10
|
+
class << self
|
11
|
+
attr_writer :service
|
12
|
+
|
13
|
+
def service
|
14
|
+
@service || "sidekiq-publisher"
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
class Subscriber
|
19
|
+
def self.subscribe_to(pattern)
|
20
|
+
ActiveSupport::Notifications.subscribe(pattern, new)
|
21
|
+
end
|
22
|
+
|
23
|
+
def finish(_name, _id, payload)
|
24
|
+
finish_span(payload)
|
25
|
+
end
|
26
|
+
|
27
|
+
private
|
28
|
+
|
29
|
+
def start_span(operation, payload, resource = nil)
|
30
|
+
resource ||= operation
|
31
|
+
payload[:datadog_span] = Datadog.tracer.trace(operation, service: service, resource: resource)
|
32
|
+
end
|
33
|
+
|
34
|
+
def start_primary_span(resource, payload)
|
35
|
+
start_span(OPERATION, payload, resource)
|
36
|
+
end
|
37
|
+
|
38
|
+
def finish_span(payload)
|
39
|
+
payload[:datadog_span]&.set_error(payload[:exception_object]) if payload.key?(:exception_object)
|
40
|
+
payload[:datadog_span]&.finish
|
41
|
+
end
|
42
|
+
|
43
|
+
def service
|
44
|
+
SidekiqPublisher::DatadogAPM.service
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
class ListenerSubscriber < Subscriber
|
49
|
+
def start(_name, _id, payload)
|
50
|
+
start_primary_span("listener.timeout", payload)
|
51
|
+
end
|
52
|
+
|
53
|
+
subscribe_to "timeout.listener.sidekiq_publisher"
|
54
|
+
end
|
55
|
+
|
56
|
+
class RunnerSubscriber < Subscriber
|
57
|
+
def start(name, _id, payload)
|
58
|
+
op_name = name.split(".").first
|
59
|
+
start_primary_span("publisher.#{op_name}", payload)
|
60
|
+
end
|
61
|
+
|
62
|
+
subscribe_to "start.publisher.sidekiq_publisher"
|
63
|
+
subscribe_to "notify.publisher.sidekiq_publisher"
|
64
|
+
subscribe_to "timeout.publisher.sidekiq_publisher"
|
65
|
+
end
|
66
|
+
|
67
|
+
class PublisherSubscriber < Subscriber
|
68
|
+
def start(name, _id, payload)
|
69
|
+
op_name = name.split(".").first
|
70
|
+
start_span("publisher.#{op_name}", payload)
|
71
|
+
end
|
72
|
+
|
73
|
+
def finish(name, id, payload)
|
74
|
+
payload[:datadog_span]&.set_tag(:published_count, payload[:published_count]) if payload.key?(:published_count)
|
75
|
+
super
|
76
|
+
end
|
77
|
+
|
78
|
+
subscribe_to "publish_batch.publisher.sidekiq_publisher"
|
79
|
+
subscribe_to "enqueue_batch.publisher.sidekiq_publisher"
|
80
|
+
end
|
81
|
+
|
82
|
+
class JobSubscriber < Subscriber
|
83
|
+
def start(_name, _id, payload)
|
84
|
+
start_span("job.purge", payload)
|
85
|
+
end
|
86
|
+
|
87
|
+
def finish(_name, _id, payload)
|
88
|
+
payload[:datadog_span]&.set_tag(:purged_count, payload[:purged_count]) if payload.key?(:purged_count)
|
89
|
+
|
90
|
+
super
|
91
|
+
end
|
92
|
+
|
93
|
+
subscribe_to "purge.job.sidekiq_publisher"
|
94
|
+
end
|
95
|
+
|
96
|
+
# This subscriber is different from the classes above because it is an ActiveSupport::Subscriber
|
97
|
+
# and responds to the error(.publisher.sidekiq_publisher) event.
|
98
|
+
class PublisherErrorSubscriber < ActiveSupport::Subscriber
|
99
|
+
def error(event)
|
100
|
+
Datadog.tracer.active_span&.set_error(event.payload[:exception_object])
|
101
|
+
end
|
102
|
+
|
103
|
+
attach_to "publisher.sidekiq_publisher"
|
104
|
+
end
|
105
|
+
end
|
106
|
+
end
|
@@ -0,0 +1,15 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "active_support/subscriber"
|
4
|
+
|
5
|
+
module SidekiqPublisher
|
6
|
+
module ExceptionReporter
|
7
|
+
class PublisherErrorSubscriber < ActiveSupport::Subscriber
|
8
|
+
def error(event)
|
9
|
+
SidekiqPublisher.exception_reporter&.call(event.payload[:exception_object])
|
10
|
+
end
|
11
|
+
|
12
|
+
attach_to "publisher.sidekiq_publisher"
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "active_support/notifications"
|
4
|
+
|
5
|
+
module SidekiqPublisher
|
6
|
+
class Instrumenter
|
7
|
+
NAMESPACE = "sidekiq_publisher"
|
8
|
+
|
9
|
+
def instrument(event_name, payload = {}, &block)
|
10
|
+
ActiveSupport::Notifications.instrument("#{event_name}.#{NAMESPACE}", payload, &block)
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -39,11 +39,12 @@ module SidekiqPublisher
|
|
39
39
|
where(id: ids).update_all(published_at: Time.now.utc)
|
40
40
|
end
|
41
41
|
|
42
|
-
def self.purge_expired_published!
|
42
|
+
def self.purge_expired_published!(instrumenter: Instrumenter.new)
|
43
43
|
SidekiqPublisher.logger.info("#{name} purging expired published jobs.")
|
44
|
-
count =
|
44
|
+
count = instrumenter.instrument("purge.job") do |notification|
|
45
|
+
notification[:purged_count] = purgeable.delete_all
|
46
|
+
end
|
45
47
|
SidekiqPublisher.logger.info("#{name} purged #{count} expired published jobs.")
|
46
|
-
SidekiqPublisher.metrics_reporter.try(:count, "sidekiq_publisher.purged", count)
|
47
48
|
end
|
48
49
|
|
49
50
|
def self.unpublished_batches(batch_size: SidekiqPublisher.batch_size)
|
@@ -0,0 +1,40 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "active_support/subscriber"
|
4
|
+
|
5
|
+
module SidekiqPublisher
|
6
|
+
module MetricsReporter
|
7
|
+
class Subscriber < ActiveSupport::Subscriber
|
8
|
+
private
|
9
|
+
|
10
|
+
def count(metric, value)
|
11
|
+
SidekiqPublisher.metrics_reporter&.try(:count, metric, value) unless value.nil?
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
class PublisherSubscriber < Subscriber
|
16
|
+
def enqueue_batch(event)
|
17
|
+
count("sidekiq_publisher.published", event.payload[:published_count])
|
18
|
+
end
|
19
|
+
|
20
|
+
attach_to "publisher.sidekiq_publisher"
|
21
|
+
end
|
22
|
+
|
23
|
+
class JobSubscriber < Subscriber
|
24
|
+
def purge(event)
|
25
|
+
count("sidekiq_publisher.purged", event.payload[:purged_count])
|
26
|
+
end
|
27
|
+
|
28
|
+
attach_to "job.sidekiq_publisher"
|
29
|
+
end
|
30
|
+
|
31
|
+
class UnpublishedSubscriber < Subscriber
|
32
|
+
def unpublished(event)
|
33
|
+
SidekiqPublisher.metrics_reporter&.
|
34
|
+
try(:gauge, "sidekiq_publisher.unpublished_count", event.payload[:unpublished_count])
|
35
|
+
end
|
36
|
+
|
37
|
+
attach_to "reporter.sidekiq_publisher"
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
@@ -5,26 +5,31 @@ require "active_support/core_ext/object/try"
|
|
5
5
|
|
6
6
|
module SidekiqPublisher
|
7
7
|
class Publisher
|
8
|
-
def initialize
|
8
|
+
def initialize(instrumenter: Instrumenter.new)
|
9
|
+
@instrumenter = instrumenter
|
9
10
|
@client = SidekiqPublisher::Client.new
|
10
11
|
@job_class_cache = {}
|
11
12
|
end
|
12
13
|
|
13
14
|
def publish
|
14
15
|
Job.unpublished_batches do |batch|
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
16
|
+
instrumenter.instrument("publish_batch.publisher") do
|
17
|
+
items = batch.map do |job|
|
18
|
+
{
|
19
|
+
"jid" => job[:job_id],
|
20
|
+
"class" => lookup_job_class(job[:job_class]),
|
21
|
+
"args" => job[:args],
|
22
|
+
"at" => job[:run_at],
|
23
|
+
"queue" => job[:queue],
|
24
|
+
"wrapped" => job[:wrapped],
|
25
|
+
"created_at" => job[:created_at].to_f,
|
26
|
+
}.tap(&:compact!)
|
27
|
+
end
|
26
28
|
|
27
|
-
|
29
|
+
instrumenter.instrument("enqueue_batch.publisher") do |notification|
|
30
|
+
enqueue_batch(batch, items, notification)
|
31
|
+
end
|
32
|
+
end
|
28
33
|
end
|
29
34
|
purge_expired_published_jobs
|
30
35
|
rescue StandardError => ex
|
@@ -33,16 +38,16 @@ module SidekiqPublisher
|
|
33
38
|
|
34
39
|
private
|
35
40
|
|
36
|
-
attr_reader :client, :job_class_cache
|
41
|
+
attr_reader :client, :job_class_cache, :instrumenter
|
37
42
|
|
38
|
-
def
|
43
|
+
def enqueue_batch(batch, items, notification)
|
39
44
|
pushed_count = client.bulk_push(items)
|
40
45
|
published_count = update_jobs_as_published!(batch)
|
41
46
|
rescue StandardError => ex
|
42
47
|
failure_warning(__method__, ex)
|
43
48
|
ensure
|
44
49
|
published_count = update_jobs_as_published!(batch) if pushed_count.present? && published_count.nil?
|
45
|
-
|
50
|
+
notification[:published_count] = published_count if published_count.present?
|
46
51
|
end
|
47
52
|
|
48
53
|
def lookup_job_class(name)
|
@@ -56,7 +61,7 @@ module SidekiqPublisher
|
|
56
61
|
end
|
57
62
|
|
58
63
|
def purge_expired_published_jobs
|
59
|
-
Job.purge_expired_published! if perform_purge?
|
64
|
+
Job.purge_expired_published!(instrumenter: instrumenter) if perform_purge?
|
60
65
|
end
|
61
66
|
|
62
67
|
def perform_purge?
|
@@ -64,17 +69,13 @@ module SidekiqPublisher
|
|
64
69
|
end
|
65
70
|
|
66
71
|
def failure_warning(method, ex)
|
67
|
-
logger.warn("#{self.class.name}: msg=\"#{method} failed\" error=#{ex.class} error_msg=#{ex.message.inspect}\n"
|
68
|
-
|
69
|
-
|
72
|
+
logger.warn("#{self.class.name}: msg=\"#{method} failed\" error=#{ex.class} error_msg=#{ex.message.inspect}\n")
|
73
|
+
instrumenter.instrument("error.publisher",
|
74
|
+
exception_object: ex, exception: [ex.class.name, ex.message])
|
70
75
|
end
|
71
76
|
|
72
77
|
def logger
|
73
78
|
SidekiqPublisher.logger
|
74
79
|
end
|
75
|
-
|
76
|
-
def metrics_reporter
|
77
|
-
SidekiqPublisher.metrics_reporter
|
78
|
-
end
|
79
80
|
end
|
80
81
|
end
|
@@ -1,11 +1,22 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require "active_record"
|
4
|
+
require "pg"
|
5
|
+
|
3
6
|
module SidekiqPublisher
|
4
7
|
module ReportUnpublishedCount
|
5
|
-
def self.call
|
6
|
-
|
7
|
-
|
8
|
-
|
8
|
+
def self.call(instrumenter: Instrumenter.new)
|
9
|
+
instrumenter.instrument("unpublished.reporter",
|
10
|
+
unpublished_count: SidekiqPublisher::Job.unpublished.count)
|
11
|
+
rescue ActiveRecord::StatementInvalid, PG::UnableToSend, PG::ConnectionBad => e
|
12
|
+
ActiveRecord::Base.clear_active_connections! if db_connection_error?(e)
|
13
|
+
raise
|
14
|
+
end
|
15
|
+
|
16
|
+
def self.db_connection_error?(error)
|
17
|
+
cause = error.is_a?(ActiveRecord::StatementInvalid) ? error.cause : error
|
18
|
+
cause.is_a?(PG::UnableToSend) || cause.is_a?(PG::ConnectionBad)
|
9
19
|
end
|
20
|
+
private_class_method :db_connection_error?
|
10
21
|
end
|
11
22
|
end
|
@@ -7,12 +7,13 @@ module SidekiqPublisher
|
|
7
7
|
LISTENER_TIMEOUT_SECONDS = 60
|
8
8
|
CHANNEL_NAME = "sidekiq_publisher_job"
|
9
9
|
|
10
|
-
def self.run
|
11
|
-
new.run
|
10
|
+
def self.run(instrumenter = Instrumenter.new)
|
11
|
+
new(instrumenter).run
|
12
12
|
end
|
13
13
|
|
14
|
-
def initialize
|
15
|
-
@
|
14
|
+
def initialize(instrumenter = Instrumenter.new)
|
15
|
+
@instrumenter = instrumenter
|
16
|
+
@publisher = Publisher.new(instrumenter: @instrumenter)
|
16
17
|
end
|
17
18
|
|
18
19
|
def run
|
@@ -20,24 +21,32 @@ module SidekiqPublisher
|
|
20
21
|
CHANNEL_NAME,
|
21
22
|
listen_timeout: LISTENER_TIMEOUT_SECONDS
|
22
23
|
) do |listener|
|
23
|
-
listener.on_start {
|
24
|
-
listener.on_notify {
|
24
|
+
listener.on_start { call_publish("start") }
|
25
|
+
listener.on_notify { call_publish("notify") }
|
25
26
|
listener.on_timeout { listener_timeout }
|
26
27
|
end
|
27
28
|
end
|
28
29
|
|
29
30
|
private
|
30
31
|
|
31
|
-
attr_reader :publisher
|
32
|
+
attr_reader :publisher, :instrumenter
|
32
33
|
|
33
|
-
def
|
34
|
-
|
35
|
-
SidekiqPublisher.logger&.warn(
|
36
|
-
"#{self.class.name}: msg='publishing pending jobs at timeout'"
|
37
|
-
)
|
34
|
+
def call_publish(event)
|
35
|
+
instrumenter.instrument("#{event}.publisher") do
|
38
36
|
publisher.publish
|
39
|
-
|
40
|
-
|
37
|
+
end
|
38
|
+
end
|
39
|
+
|
40
|
+
def listener_timeout
|
41
|
+
instrumenter.instrument("timeout.listener") do
|
42
|
+
if Job.unpublished.exists?
|
43
|
+
SidekiqPublisher.logger&.warn(
|
44
|
+
"#{self.class.name}: msg='publishing pending jobs at timeout'"
|
45
|
+
)
|
46
|
+
call_publish("timeout")
|
47
|
+
else
|
48
|
+
Job.purge_expired_published!(instrumenter: instrumenter)
|
49
|
+
end
|
41
50
|
end
|
42
51
|
end
|
43
52
|
end
|
data/sidekiq_publisher.gemspec
CHANGED
@@ -45,6 +45,7 @@ Gem::Specification.new do |spec|
|
|
45
45
|
spec.add_development_dependency "appraisal"
|
46
46
|
spec.add_development_dependency "bundler", "~> 1.12"
|
47
47
|
spec.add_development_dependency "database_cleaner"
|
48
|
+
spec.add_development_dependency "ddtrace", ">= 0.39.0"
|
48
49
|
spec.add_development_dependency "ezcater_matchers"
|
49
50
|
spec.add_development_dependency "ezcater_rubocop", "1.0.2"
|
50
51
|
spec.add_development_dependency "factory_bot"
|
@@ -57,6 +58,6 @@ Gem::Specification.new do |spec|
|
|
57
58
|
spec.add_development_dependency "simplecov", "< 0.18"
|
58
59
|
|
59
60
|
spec.add_runtime_dependency "activerecord-postgres_pub_sub", ">= 0.4.0"
|
60
|
-
spec.add_runtime_dependency "activesupport", ">= 5.1", "< 6.
|
61
|
+
spec.add_runtime_dependency "activesupport", ">= 5.1", "< 6.2"
|
61
62
|
spec.add_runtime_dependency "sidekiq", ">= 5.0.4", "< 7.0"
|
62
63
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq_publisher
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.8.0.rc0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- ezCater, Inc
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2021-03-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -66,6 +66,20 @@ dependencies:
|
|
66
66
|
- - ">="
|
67
67
|
- !ruby/object:Gem::Version
|
68
68
|
version: '0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: ddtrace
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: 0.39.0
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: 0.39.0
|
69
83
|
- !ruby/object:Gem::Dependency
|
70
84
|
name: ezcater_matchers
|
71
85
|
requirement: !ruby/object:Gem::Requirement
|
@@ -229,7 +243,7 @@ dependencies:
|
|
229
243
|
version: '5.1'
|
230
244
|
- - "<"
|
231
245
|
- !ruby/object:Gem::Version
|
232
|
-
version: '6.
|
246
|
+
version: '6.2'
|
233
247
|
type: :runtime
|
234
248
|
prerelease: false
|
235
249
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -239,7 +253,7 @@ dependencies:
|
|
239
253
|
version: '5.1'
|
240
254
|
- - "<"
|
241
255
|
- !ruby/object:Gem::Version
|
242
|
-
version: '6.
|
256
|
+
version: '6.2'
|
243
257
|
- !ruby/object:Gem::Dependency
|
244
258
|
name: sidekiq
|
245
259
|
requirement: !ruby/object:Gem::Requirement
|
@@ -268,11 +282,14 @@ extensions: []
|
|
268
282
|
extra_rdoc_files: []
|
269
283
|
files:
|
270
284
|
- ".codeclimate.yml"
|
285
|
+
- ".github/CODEOWNERS"
|
271
286
|
- Appraisals
|
272
287
|
- CHANGELOG.md
|
288
|
+
- Dockerfile
|
273
289
|
- Gemfile
|
274
290
|
- LICENSE.txt
|
275
291
|
- README.md
|
292
|
+
- docker-compose.yml
|
276
293
|
- gemfiles/rails_5.1.gemfile
|
277
294
|
- gemfiles/rails_5.2.gemfile
|
278
295
|
- gemfiles/rails_5.2_sidekiq_5.0.gemfile
|
@@ -283,12 +300,19 @@ files:
|
|
283
300
|
- gemfiles/rails_6.0_sidekiq_5.2.gemfile
|
284
301
|
- gemfiles/rails_6.0_sidekiq_6.0.gemfile
|
285
302
|
- gemfiles/rails_6.0_sidekiq_6.1.gemfile
|
303
|
+
- gemfiles/rails_6.1_sidekiq_5.2.gemfile
|
304
|
+
- gemfiles/rails_6.1_sidekiq_6.0.gemfile
|
305
|
+
- gemfiles/rails_6.1_sidekiq_6.1.gemfile
|
286
306
|
- lib/active_job/queue_adapters/sidekiq_publisher_adapter.rb
|
287
307
|
- lib/generators/sidekiq_publisher/install_generator.rb
|
288
308
|
- lib/generators/sidekiq_publisher/templates/create_sidekiq_publisher_jobs.rb
|
289
309
|
- lib/sidekiq_publisher.rb
|
290
310
|
- lib/sidekiq_publisher/client.rb
|
311
|
+
- lib/sidekiq_publisher/datadog_apm.rb
|
312
|
+
- lib/sidekiq_publisher/exception_reporter.rb
|
313
|
+
- lib/sidekiq_publisher/instrumenter.rb
|
291
314
|
- lib/sidekiq_publisher/job.rb
|
315
|
+
- lib/sidekiq_publisher/metrics_reporter.rb
|
292
316
|
- lib/sidekiq_publisher/publisher.rb
|
293
317
|
- lib/sidekiq_publisher/railtie.rb
|
294
318
|
- lib/sidekiq_publisher/report_unpublished_count.rb
|
@@ -315,11 +339,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
315
339
|
version: '0'
|
316
340
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
317
341
|
requirements:
|
318
|
-
- - "
|
342
|
+
- - ">"
|
319
343
|
- !ruby/object:Gem::Version
|
320
|
-
version:
|
344
|
+
version: 1.3.1
|
321
345
|
requirements: []
|
322
|
-
|
346
|
+
rubyforge_project:
|
347
|
+
rubygems_version: 2.7.6.2
|
323
348
|
signing_key:
|
324
349
|
specification_version: 4
|
325
350
|
summary: Publisher for enqueuing jobs to Sidekiq
|