propono 2.0.0.rc3 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.github/workflows/tests.yml +30 -0
- data/.ruby-version +1 -0
- data/CHANGELOG.md +9 -0
- data/README.md +50 -24
- data/docs/upgrading.md +48 -0
- data/lib/propono/components/aws_client.rb +7 -5
- data/lib/propono/components/aws_config.rb +7 -13
- data/lib/propono/components/client.rb +20 -15
- data/lib/propono/components/queue_subscription.rb +1 -1
- data/lib/propono/configuration.rb +10 -8
- data/lib/propono/services/queue_listener.rb +6 -5
- data/lib/propono/version.rb +1 -1
- data/propono.gemspec +4 -4
- data/test/components/aws_config_test.rb +12 -32
- data/test/components/client_test.rb +21 -4
- data/test/config.yml.example +22 -3
- data/test/configuration_test.rb +25 -40
- data/test/integration/integration_test.rb +1 -3
- data/test/services/queue_listener_test.rb +27 -1
- data/test/test_helper.rb +0 -3
- metadata +17 -17
- data/.travis.yml +0 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: a7790d050fa3ee543affb106f6ef8478f7576c07270a7c285ffb3dc60dc65635
|
4
|
+
data.tar.gz: 9fab2edbfda2def0ce81b27ff12a9da4d57ecc3b4e45fded5d1513b2a02b855e
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: beb2e62ada80b404c4b590b88e2166034217f812483c097e1d1b86585076c261e7bc6b947a1a6e6cfdf3af362c63cdeeb9760d85c3c4dcfed5016b18b14bdd5b
|
7
|
+
data.tar.gz: 694302c72bd5aaada4f5472707f52c491d13d2707c7cc3d69575c9330961cd8049e5dcb4baf1c08f24ed5b9d8ae07173849995c922eae559a6746ee806460efb
|
@@ -0,0 +1,30 @@
|
|
1
|
+
name: Tests
|
2
|
+
|
3
|
+
on:
|
4
|
+
push:
|
5
|
+
branches: [main]
|
6
|
+
pull_request:
|
7
|
+
branches: [main]
|
8
|
+
|
9
|
+
jobs:
|
10
|
+
test:
|
11
|
+
name: Ruby ${{ matrix.ruby-version }} - ${{ matrix.os }} - ${{ github.event_name }}
|
12
|
+
runs-on: ${{ matrix.os }}
|
13
|
+
strategy:
|
14
|
+
fail-fast: false
|
15
|
+
matrix:
|
16
|
+
os:
|
17
|
+
- ubuntu-latest
|
18
|
+
ruby-version: [2.6, 2.7]
|
19
|
+
|
20
|
+
steps:
|
21
|
+
- uses: actions/checkout@v2
|
22
|
+
|
23
|
+
- name: Set up Ruby
|
24
|
+
uses: ruby/setup-ruby@a699edbce608a2c128dedad88e3b6a0e28687b3c
|
25
|
+
with:
|
26
|
+
ruby-version: ${{ matrix.ruby-version }}
|
27
|
+
bundler-cache: true
|
28
|
+
|
29
|
+
- name: Test
|
30
|
+
run: bundle exec rake test:local
|
data/.ruby-version
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
2.6.6
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,12 @@
|
|
1
|
+
# 3.0.0 / 2021-01-10
|
2
|
+
* [FEATURE] Improve AWS configuration.
|
3
|
+
|
4
|
+
# 2.2.0 / 2019-09-21
|
5
|
+
* [FEATURE] Add setting to disable slow queue
|
6
|
+
|
7
|
+
# 2.1.0 / 2017-03-14
|
8
|
+
* [FEATURE] Added visibility_timeout to listen
|
9
|
+
|
1
10
|
# 2.0.0 / 2017-03-14
|
2
11
|
* [FEATURE] Remove UDP and TCP support
|
3
12
|
* [FEATURE] Change default publish behaviour from async to sync
|
data/README.md
CHANGED
@@ -21,13 +21,9 @@ Propono::Client.new.publish('some-topic', "The Best Message Ever")
|
|
21
21
|
# - "I just received The Best Message Ever"
|
22
22
|
```
|
23
23
|
|
24
|
-
##
|
24
|
+
## Upgrading
|
25
25
|
|
26
|
-
|
27
|
-
- We moved from a global interface to a client interface. Rather than calling `publish` and equivalent on `Propono`, you should now initialize a `Propono::Client` and then call everything on that client. This fixes issues with thread safety and global config.
|
28
|
-
- We have also removed the dependancy on Fog and instead switch to the `sns` and `sqs` mini-gems of `aws-sdk`.
|
29
|
-
- UDP and TCP support have been removed, and `subscribe_by_post` has been removed.
|
30
|
-
- We are now using long-polling. This makes Propono **significantly** faster (10-100x).
|
26
|
+
Upgrades from v1 to v2, and v2 to v3 are covered in the [upgrade documentation](docs/upgrading.md).
|
31
27
|
|
32
28
|
## Installation
|
33
29
|
|
@@ -41,18 +37,12 @@ And then execute:
|
|
41
37
|
|
42
38
|
## Usage
|
43
39
|
|
44
|
-
The first thing to do is setup some configuration
|
40
|
+
The first thing to do is setup some configuration for Propono.
|
41
|
+
It's best to do this in an initializer, or at the start of your application.
|
42
|
+
If you need to setup AWS authentication, see the [AWS Configuration](#aws-configuration) section.
|
45
43
|
|
46
44
|
```ruby
|
47
45
|
client = Propono::Client.new
|
48
|
-
client.config.access_key = "access-key" # From AWS
|
49
|
-
client.config.secret_key = "secret-key" # From AWS
|
50
|
-
client.config.queue_region = "queue-region" # From AWS
|
51
|
-
|
52
|
-
# Or use the IAM profile of the machine
|
53
|
-
client.config.use_iam_profile = true
|
54
|
-
client.config.queue_region = "queue-region" # From AWS
|
55
|
-
|
56
46
|
```
|
57
47
|
|
58
48
|
You can then start publishing messages easily from anywhere in your codebase.
|
@@ -81,28 +71,64 @@ This is because a queue is established for each application_name/topic combinati
|
|
81
71
|
* subscribers that share the same `application_name` will act as multiple workers on the same queue. Only one will get to process each message.
|
82
72
|
* subscribers that have a different `application_name` will each get a copy of a message to process independently i.e. acts as a one-to-many broadcast.
|
83
73
|
|
84
|
-
### Configuration
|
74
|
+
### AWS Configuration
|
75
|
+
|
76
|
+
By default, Propono will create SQS and SNS clients with no options.
|
77
|
+
In the absence of options, these clients will make use of the credentials on the current host.
|
78
|
+
See the [AWS SDK For Ruby Configuration documentation](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html) for more details.
|
79
|
+
|
80
|
+
To manually configure options for use with AWS, use `aws_options`, which sets options to be passed to both clients. For example:
|
81
|
+
|
82
|
+
client = Propono::Client.new do |config|
|
83
|
+
config.aws_options = {
|
84
|
+
region: 'aws_region',
|
85
|
+
access_key_id: 'your_access_key_id',
|
86
|
+
secret_access_key: 'your_secret_access_key'
|
87
|
+
}
|
88
|
+
end
|
85
89
|
|
86
|
-
|
90
|
+
In addition to this, there are also `sqs_options` and `sns_options`, used to configure each client independently.
|
91
|
+
See the [SQS Client](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/SQS/Client.html#initialize-instance_method) and [SNS Client](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/SNS/Client.html#initialize-instance_method) documentation for available options.
|
92
|
+
These individual options are merged with `aws_options` with the per-client options taking precendence.
|
93
|
+
|
94
|
+
### General Configuration
|
87
95
|
|
88
96
|
```
|
89
97
|
Propono::Client.new do |config|
|
90
|
-
#
|
91
|
-
config.
|
92
|
-
config.
|
93
|
-
|
94
|
-
config.use_iam_profile = true
|
98
|
+
# AWS Configuration, see above.
|
99
|
+
config.aws_options = {...}
|
100
|
+
config.sqs_options = {...}
|
101
|
+
config.sns_options = {...}
|
95
102
|
|
96
|
-
config.queue_region = "An AWS queue region"
|
97
103
|
config.application_name = "A name unique in your network"
|
98
104
|
config.logger = "A logger such as Log4r or Rails.logger"
|
99
105
|
|
100
106
|
config.max_retries = "The number of retries if a message raises an exception before being placed on the failed queue"
|
101
107
|
config.num_messages_per_poll = "The number of messages retrieved per poll to SQS"
|
108
|
+
|
109
|
+
config.slow_queue_enabled = true
|
110
|
+
end
|
111
|
+
```
|
112
|
+
|
113
|
+
### Options
|
114
|
+
|
115
|
+
#### Async
|
116
|
+
|
117
|
+
By default messages are posted inline, blocking the main thread. The `async: true` option can be sent when posting a message, which will spawn a new thread for the message networking calls, and unblocking the main thread.
|
118
|
+
|
119
|
+
#### Visiblity Timeout
|
120
|
+
|
121
|
+
For certain tasks (e.g. video processing), being able to hold messages for longer is important. To achieve this, the [visibility timeout of a message](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) can be changed on the call to listen. e.g.
|
122
|
+
|
123
|
+
```
|
124
|
+
client.listen('long-running-tasks', visiblity_timeout: 3600) do |message|
|
125
|
+
puts "I just received: #{message}"
|
102
126
|
end
|
103
127
|
```
|
104
128
|
|
105
|
-
|
129
|
+
### Slow Queue
|
130
|
+
|
131
|
+
The slow queue can be disabled by setting `slow_queue_enabled` to `false`. This will yield performance improvements if you do not make use of the "slow queue" functionality.
|
106
132
|
|
107
133
|
### Is it any good?
|
108
134
|
|
data/docs/upgrading.md
ADDED
@@ -0,0 +1,48 @@
|
|
1
|
+
# Upgrading
|
2
|
+
|
3
|
+
## Changes from v2 to v3
|
4
|
+
|
5
|
+
Version 3 changed the way configuration options for the two AWS services are
|
6
|
+
passed to the client gems. Instead of Propono attempting to guess which
|
7
|
+
configuration options you might want, it now accepts hashes for AWS
|
8
|
+
configuration which are passed directly to the appropriate clients.
|
9
|
+
|
10
|
+
If you are upgrading from v2 to v3, and using the configuration as previously
|
11
|
+
given in the README, you need to change from:
|
12
|
+
|
13
|
+
```ruby
|
14
|
+
client = Propono::Client.new
|
15
|
+
client.config.queue_region = 'aws_region'
|
16
|
+
client.config.access_key = 'your_access_key_id'
|
17
|
+
client.config.secret_key = 'your_secret_access_key'
|
18
|
+
```
|
19
|
+
|
20
|
+
To:
|
21
|
+
|
22
|
+
```ruby
|
23
|
+
client = Propono::Client.new do |config|
|
24
|
+
config.aws_options = {
|
25
|
+
region: 'aws_region',
|
26
|
+
access_key_id: 'your_access_key_id',
|
27
|
+
secret_access_key: 'your_secret_access_key'
|
28
|
+
}
|
29
|
+
end
|
30
|
+
```
|
31
|
+
|
32
|
+
For a full rundown, see the [AWS Configuration
|
33
|
+
section](../README.md#aws-configuration) of the README.
|
34
|
+
|
35
|
+
|
36
|
+
## Changes from v1 to v2
|
37
|
+
|
38
|
+
Version 2 of Propono changed a few things:
|
39
|
+
- We moved from a global interface to a client interface. Rather than calling
|
40
|
+
`publish` and equivalent on `Propono`, you should now initialize a
|
41
|
+
`Propono::Client` and then call everything on that client. This fixes issues
|
42
|
+
with thread safety and global config.
|
43
|
+
- We have also removed the dependancy on Fog and instead switch to the `sns`
|
44
|
+
and `sqs` mini-gems of `aws-sdk`.
|
45
|
+
- UDP and TCP support have been removed, and `subscribe_by_post` has been
|
46
|
+
removed.
|
47
|
+
- We are now using long-polling. This makes Propono **significantly** faster
|
48
|
+
(10-100x).
|
@@ -43,16 +43,18 @@ module Propono
|
|
43
43
|
def set_sqs_policy(queue, policy)
|
44
44
|
sqs_client.set_queue_attributes(
|
45
45
|
queue_url: queue.url,
|
46
|
-
attributes: {
|
46
|
+
attributes: { Policy: policy }
|
47
47
|
)
|
48
48
|
end
|
49
49
|
|
50
|
-
def read_from_sqs(queue, num_messages, long_poll: true)
|
50
|
+
def read_from_sqs(queue, num_messages, long_poll: true, visibility_timeout: nil)
|
51
51
|
wait_time_seconds = long_poll ? 20 : 0
|
52
|
+
visibility_timeout ||= 30
|
52
53
|
sqs_client.receive_message(
|
53
54
|
queue_url: queue.url,
|
54
55
|
wait_time_seconds: wait_time_seconds,
|
55
|
-
max_number_of_messages: num_messages
|
56
|
+
max_number_of_messages: num_messages,
|
57
|
+
visibility_timeout: visibility_timeout
|
56
58
|
).messages
|
57
59
|
end
|
58
60
|
|
@@ -66,11 +68,11 @@ module Propono
|
|
66
68
|
private
|
67
69
|
|
68
70
|
def sns_client
|
69
|
-
@sns_client ||= Aws::SNS::Client.new(aws_config.
|
71
|
+
@sns_client ||= Aws::SNS::Client.new(aws_config.sns_options)
|
70
72
|
end
|
71
73
|
|
72
74
|
def sqs_client
|
73
|
-
@sqs_client ||= Aws::SQS::Client.new(aws_config.
|
75
|
+
@sqs_client ||= Aws::SQS::Client.new(aws_config.sqs_options)
|
74
76
|
end
|
75
77
|
end
|
76
78
|
end
|
@@ -5,19 +5,13 @@ module Propono
|
|
5
5
|
@config = config
|
6
6
|
end
|
7
7
|
|
8
|
-
def
|
9
|
-
|
10
|
-
{
|
11
|
-
:use_iam_profile => true,
|
12
|
-
:region => @config.queue_region
|
13
|
-
}
|
14
|
-
else
|
15
|
-
{
|
16
|
-
:access_key_id => @config.access_key,
|
17
|
-
:secret_access_key => @config.secret_key,
|
18
|
-
:region => @config.queue_region
|
19
|
-
}
|
20
|
-
end
|
8
|
+
def sqs_options
|
9
|
+
@config.aws_options.merge(@config.sqs_options)
|
21
10
|
end
|
11
|
+
|
12
|
+
def sns_options
|
13
|
+
@config.aws_options.merge(@config.sns_options)
|
14
|
+
end
|
15
|
+
|
22
16
|
end
|
23
17
|
end
|
@@ -1,23 +1,28 @@
|
|
1
1
|
module Propono
|
2
2
|
class Client
|
3
3
|
|
4
|
-
# Propono configuration
|
4
|
+
# Propono configuration.
|
5
5
|
#
|
6
6
|
# Settings should be set in an initializer or using some
|
7
|
-
# other method that
|
8
|
-
# Propono code is used.
|
7
|
+
# other method that ensures they are set before any
|
8
|
+
# Propono code is used.
|
9
9
|
#
|
10
|
-
#
|
10
|
+
# They can be set in one of the following ways:
|
11
11
|
#
|
12
|
-
#
|
12
|
+
# 1. As options passed to <tt>new</tt> as a hash.
|
13
13
|
#
|
14
|
-
#
|
15
|
-
#
|
16
|
-
#
|
17
|
-
#
|
18
|
-
#
|
19
|
-
#
|
20
|
-
#
|
14
|
+
# Propono::Client.new(application_name: 'my-application')
|
15
|
+
#
|
16
|
+
# 2. As options passed to <tt>new</tt> using a block.
|
17
|
+
#
|
18
|
+
# Propono::Client.new do |config"
|
19
|
+
# config.application_name: 'my-application'
|
20
|
+
# end
|
21
|
+
#
|
22
|
+
# 3. By calling the <tt>Propono::Client#configure</tt>.
|
23
|
+
# client.configure do |config|
|
24
|
+
# config.access_key = "my-access-key"
|
25
|
+
# end
|
21
26
|
|
22
27
|
attr_reader :config, :aws_client
|
23
28
|
def initialize(settings = {}, &block)
|
@@ -33,7 +38,7 @@ module Propono
|
|
33
38
|
@aws_client = AwsClient.new(AwsConfig.new(config))
|
34
39
|
end
|
35
40
|
|
36
|
-
def configure
|
41
|
+
def configure
|
37
42
|
yield config
|
38
43
|
end
|
39
44
|
|
@@ -69,8 +74,8 @@ module Propono
|
|
69
74
|
#
|
70
75
|
# @param [String] topic The topic to subscribe to.
|
71
76
|
# @param &message_processor The block to yield for each message.
|
72
|
-
def listen(topic_name, &message_processor)
|
73
|
-
QueueListener.listen(aws_client, config, topic_name, &message_processor)
|
77
|
+
def listen(topic_name, options = {}, &message_processor)
|
78
|
+
QueueListener.listen(aws_client, config, topic_name, options, &message_processor)
|
74
79
|
end
|
75
80
|
|
76
81
|
# Listens on a queue and yields for each message
|
@@ -15,7 +15,7 @@ module Propono
|
|
15
15
|
@topic_name = topic_name
|
16
16
|
@suffixed_topic_name = "#{topic_name}#{propono_config.queue_suffix}"
|
17
17
|
@suffixed_slow_topic_name = "#{topic_name}#{propono_config.queue_suffix}-slow"
|
18
|
-
@queue_name = "#{propono_config.application_name.
|
18
|
+
@queue_name = "#{propono_config.application_name.tr(" ", "_")}-#{@suffixed_topic_name}"
|
19
19
|
end
|
20
20
|
|
21
21
|
def create
|
@@ -15,24 +15,26 @@ module Propono
|
|
15
15
|
end
|
16
16
|
end
|
17
17
|
|
18
|
-
add_setting :
|
19
|
-
add_setting :
|
20
|
-
add_setting :
|
18
|
+
add_setting :aws_options
|
19
|
+
add_setting :sqs_options
|
20
|
+
add_setting :sns_options
|
21
21
|
add_setting :application_name
|
22
22
|
add_setting :logger
|
23
23
|
add_setting :max_retries
|
24
24
|
add_setting :num_messages_per_poll
|
25
|
-
|
26
|
-
add_setting :
|
27
|
-
add_setting :queue_suffix, required: false
|
25
|
+
add_setting :slow_queue_enabled, required: false
|
26
|
+
add_setting :queue_suffix, required: false
|
28
27
|
|
29
28
|
def initialize
|
30
29
|
@settings = {
|
30
|
+
aws_options: {},
|
31
|
+
sqs_options: {},
|
32
|
+
sns_options: {},
|
31
33
|
logger: Propono::Logger.new,
|
32
34
|
queue_suffix: "",
|
33
|
-
use_iam_profile: false,
|
34
35
|
max_retries: 0,
|
35
|
-
num_messages_per_poll:
|
36
|
+
num_messages_per_poll: 1,
|
37
|
+
slow_queue_enabled: true
|
36
38
|
}
|
37
39
|
end
|
38
40
|
|
@@ -9,12 +9,13 @@ module Propono
|
|
9
9
|
new(*args, &message_processor).drain
|
10
10
|
end
|
11
11
|
|
12
|
-
attr_reader :aws_client, :propono_config, :topic_name, :message_processor
|
13
|
-
def initialize(aws_client, propono_config, topic_name, &message_processor)
|
12
|
+
attr_reader :aws_client, :propono_config, :topic_name, :visibility_timeout, :message_processor
|
13
|
+
def initialize(aws_client, propono_config, topic_name, options = {}, &message_processor)
|
14
14
|
@aws_client = aws_client
|
15
15
|
@propono_config = propono_config
|
16
16
|
@topic_name = topic_name
|
17
17
|
@message_processor = message_processor
|
18
|
+
@visibility_timeout = options[:visibility_timeout] || nil
|
18
19
|
end
|
19
20
|
|
20
21
|
def listen
|
@@ -27,18 +28,18 @@ module Propono
|
|
27
28
|
def drain
|
28
29
|
raise ProponoError.new("topic_name is nil") unless topic_name
|
29
30
|
true while read_messages_from_queue(main_queue, 10, long_poll: false)
|
30
|
-
true while read_messages_from_queue(slow_queue, 10, long_poll: false)
|
31
|
+
true while read_messages_from_queue(slow_queue, 10, long_poll: false) if propono_config.slow_queue_enabled
|
31
32
|
end
|
32
33
|
|
33
34
|
private
|
34
35
|
|
35
36
|
def read_messages
|
36
37
|
read_messages_from_queue(main_queue, propono_config.num_messages_per_poll) ||
|
37
|
-
|
38
|
+
(propono_config.slow_queue_enabled ? read_messages_from_queue(slow_queue, 1) : nil)
|
38
39
|
end
|
39
40
|
|
40
41
|
def read_messages_from_queue(queue, num_messages, long_poll: true)
|
41
|
-
messages = aws_client.read_from_sqs(queue, num_messages, long_poll: long_poll)
|
42
|
+
messages = aws_client.read_from_sqs(queue, num_messages, long_poll: long_poll, visibility_timeout: visibility_timeout)
|
42
43
|
if messages.empty?
|
43
44
|
false
|
44
45
|
else
|
data/lib/propono/version.rb
CHANGED
data/propono.gemspec
CHANGED
@@ -6,11 +6,11 @@ require 'propono/version'
|
|
6
6
|
Gem::Specification.new do |spec|
|
7
7
|
spec.name = "propono"
|
8
8
|
spec.version = Propono::VERSION
|
9
|
-
spec.authors = ["
|
10
|
-
spec.email = ["jez.walker@gmail.com"
|
9
|
+
spec.authors = ["iHiD", "dougal", "ccare", "MalcyL"]
|
10
|
+
spec.email = ["jez.walker@gmail.com"]
|
11
11
|
spec.description = %q{Pub / Sub Library using Amazon Web Services}
|
12
12
|
spec.summary = %q{General purpose pub/sub library built on top of AWS SNS and SQS}
|
13
|
-
spec.homepage = ""
|
13
|
+
spec.homepage = "https://github.com/iHiD/propono/"
|
14
14
|
spec.license = "MIT"
|
15
15
|
|
16
16
|
spec.files = `git ls-files`.split($/)
|
@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
|
|
21
21
|
spec.add_dependency "aws-sdk-sns"
|
22
22
|
spec.add_dependency "aws-sdk-sqs"
|
23
23
|
|
24
|
-
spec.add_development_dependency "bundler", "~> 1
|
24
|
+
spec.add_development_dependency "bundler", "~> 2.1"
|
25
25
|
spec.add_development_dependency "rake"
|
26
26
|
spec.add_development_dependency "mocha"
|
27
27
|
spec.add_development_dependency "yard"
|
@@ -7,47 +7,27 @@ module Propono
|
|
7
7
|
super
|
8
8
|
@config = Propono::Configuration.new
|
9
9
|
|
10
|
-
@config.
|
11
|
-
@config.
|
12
|
-
@config.
|
10
|
+
@config.aws_options = { a: 'any', b: 'aws-specific' }
|
11
|
+
@config.sqs_options = { a: 'sqs', c: 'sqs-specific' }
|
12
|
+
@config.sns_options = { a: 'sns', c: 'sns-specific' }
|
13
13
|
|
14
14
|
@aws_config = Propono::AwsConfig.new(@config)
|
15
15
|
end
|
16
16
|
|
17
|
-
def
|
18
|
-
assert_equal
|
17
|
+
def test_overwritten_keys_take_precendence
|
18
|
+
assert_equal 'sqs', @aws_config.sqs_options[:a]
|
19
|
+
assert_equal 'sns', @aws_config.sns_options[:a]
|
19
20
|
end
|
20
21
|
|
21
|
-
def
|
22
|
-
assert_equal
|
22
|
+
def test_common_keys_remain
|
23
|
+
assert_equal 'aws-specific', @aws_config.sqs_options[:b]
|
24
|
+
assert_equal 'aws-specific', @aws_config.sns_options[:b]
|
23
25
|
end
|
24
26
|
|
25
|
-
def
|
26
|
-
assert_equal
|
27
|
+
def test_specific_keys_remain
|
28
|
+
assert_equal 'sqs-specific', @aws_config.sqs_options[:c]
|
29
|
+
assert_equal 'sns-specific', @aws_config.sns_options[:c]
|
27
30
|
end
|
28
31
|
|
29
|
-
def test_no_iam_profile_selected
|
30
|
-
assert ! @aws_config.aws_options.has_key?(:use_iam_profile)
|
31
|
-
end
|
32
|
-
|
33
|
-
def test_use_iam_profile
|
34
|
-
@config.use_iam_profile = true
|
35
|
-
assert @aws_config.aws_options[:use_iam_profile]
|
36
|
-
end
|
37
|
-
|
38
|
-
def test_selecting_use_iam_profile_results_in_no_access_key
|
39
|
-
@config.use_iam_profile = true
|
40
|
-
assert ! @aws_config.aws_options.has_key?(:access_key_id)
|
41
|
-
end
|
42
|
-
|
43
|
-
def test_selecting_use_iam_profile_results_in_no_secret_key
|
44
|
-
@config.use_iam_profile = true
|
45
|
-
assert ! @aws_config.aws_options.has_key?(:secret_access_key)
|
46
|
-
end
|
47
|
-
|
48
|
-
def test_region_when_using_iam_profile
|
49
|
-
@config.use_iam_profile = true
|
50
|
-
assert_equal "test-queue-region", @aws_config.aws_options[:region]
|
51
|
-
end
|
52
32
|
end
|
53
33
|
end
|
@@ -40,11 +40,28 @@ module Propono
|
|
40
40
|
QueueListener.expects(:listen).with(
|
41
41
|
client.aws_client,
|
42
42
|
client.config,
|
43
|
-
topic
|
43
|
+
topic,
|
44
|
+
{}
|
44
45
|
)
|
45
46
|
client.listen(topic)
|
46
47
|
end
|
47
48
|
|
49
|
+
def test_listen_calls_queue_listener_with_options
|
50
|
+
topic = 'foobar'
|
51
|
+
options = {foo: 'bar'}
|
52
|
+
|
53
|
+
client = Propono::Client.new
|
54
|
+
QueueListener.expects(:listen).with(
|
55
|
+
client.aws_client,
|
56
|
+
client.config,
|
57
|
+
topic,
|
58
|
+
options
|
59
|
+
)
|
60
|
+
client.listen(topic, options)
|
61
|
+
end
|
62
|
+
|
63
|
+
|
64
|
+
|
48
65
|
def test_drain_queue_calls_queue_listener
|
49
66
|
topic = 'foobar'
|
50
67
|
|
@@ -58,11 +75,11 @@ module Propono
|
|
58
75
|
end
|
59
76
|
|
60
77
|
def test_block_configuration_syntax
|
61
|
-
|
78
|
+
test_application_name = "my-application"
|
62
79
|
client = Propono::Client.new do |config|
|
63
|
-
config.
|
80
|
+
config.application_name = test_application_name
|
64
81
|
end
|
65
|
-
assert_equal
|
82
|
+
assert_equal test_application_name, client.config.application_name
|
66
83
|
end
|
67
84
|
end
|
68
85
|
end
|
data/test/config.yml.example
CHANGED
@@ -1,4 +1,23 @@
|
|
1
|
-
access_key: test-aws-access-key
|
2
|
-
secret_key: test-aws-secret-key
|
3
|
-
queue_region: test-aws-region
|
4
1
|
application_name: tests-yourinitials
|
2
|
+
|
3
|
+
# Whatever keys are in aws_options are passed directly to the AWS clients.
|
4
|
+
|
5
|
+
# Option 1 - Do nothing.
|
6
|
+
# AWS clients will either use the default profile in ~/.aws/credentials, or use an IAM Role if on EC2.
|
7
|
+
|
8
|
+
# Option 2 - Use environment variables
|
9
|
+
# You can set a non-default local profile with the AWS_PROFILE environment variable.
|
10
|
+
# You can also set keys directly: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html#aws-ruby-sdk-credentials-environment
|
11
|
+
|
12
|
+
# Option 3 - Set values directly.
|
13
|
+
aws_options:
|
14
|
+
# Required
|
15
|
+
region: 'test-aws-region'
|
16
|
+
|
17
|
+
# Set keys:
|
18
|
+
# access_key_id: test-aws-access-key
|
19
|
+
# secret_access_key: test-aws-secret-key
|
20
|
+
|
21
|
+
# Or set a profile:
|
22
|
+
# profile: profile-name
|
23
|
+
|
data/test/configuration_test.rb
CHANGED
@@ -12,37 +12,40 @@ module Propono
|
|
12
12
|
refute propono_config.nil?
|
13
13
|
end
|
14
14
|
|
15
|
-
def
|
16
|
-
|
15
|
+
def test_application_name
|
16
|
+
application_name = "test-application-name"
|
17
|
+
propono_config.application_name = application_name
|
18
|
+
assert_equal application_name, propono_config.application_name
|
17
19
|
end
|
18
20
|
|
19
|
-
def
|
20
|
-
propono_config.
|
21
|
-
assert propono_config.use_iam_profile
|
21
|
+
def test_default_aws_options
|
22
|
+
assert_equal({}, propono_config.aws_options)
|
22
23
|
end
|
23
24
|
|
24
|
-
def
|
25
|
-
|
26
|
-
propono_config.
|
27
|
-
assert_equal
|
25
|
+
def test_aws_options
|
26
|
+
opts = { foo: 'bar' }
|
27
|
+
propono_config.aws_options = opts
|
28
|
+
assert_equal opts, propono_config.aws_options
|
28
29
|
end
|
29
30
|
|
30
|
-
def
|
31
|
-
|
32
|
-
propono_config.secret_key = secret_key
|
33
|
-
assert_equal secret_key, propono_config.secret_key
|
31
|
+
def test_default_sqs_options
|
32
|
+
assert_equal({}, propono_config.sqs_options)
|
34
33
|
end
|
35
34
|
|
36
|
-
def
|
37
|
-
|
38
|
-
propono_config.
|
39
|
-
assert_equal
|
35
|
+
def test_sqs_options
|
36
|
+
opts = { foo: 'bar' }
|
37
|
+
propono_config.sqs_options = opts
|
38
|
+
assert_equal opts, propono_config.sqs_options
|
40
39
|
end
|
41
40
|
|
42
|
-
def
|
43
|
-
|
44
|
-
|
45
|
-
|
41
|
+
def test_default_sns_options
|
42
|
+
assert_equal({}, propono_config.sns_options)
|
43
|
+
end
|
44
|
+
|
45
|
+
def test_sns_options
|
46
|
+
opts = { foo: 'bar' }
|
47
|
+
propono_config.sns_options = opts
|
48
|
+
assert_equal opts, propono_config.sns_options
|
46
49
|
end
|
47
50
|
|
48
51
|
def test_default_logger
|
@@ -65,7 +68,7 @@ module Propono
|
|
65
68
|
end
|
66
69
|
|
67
70
|
def test_default_num_messages_per_poll
|
68
|
-
assert_equal
|
71
|
+
assert_equal 1, propono_config.num_messages_per_poll
|
69
72
|
end
|
70
73
|
|
71
74
|
def test_num_messages_per_poll
|
@@ -74,24 +77,6 @@ module Propono
|
|
74
77
|
assert_equal val, propono_config.num_messages_per_poll
|
75
78
|
end
|
76
79
|
|
77
|
-
def test_missing_access_key_throws_exception
|
78
|
-
assert_raises(ProponoConfigurationError) do
|
79
|
-
propono_config.access_key
|
80
|
-
end
|
81
|
-
end
|
82
|
-
|
83
|
-
def test_missing_secret_key_throws_exception
|
84
|
-
assert_raises(ProponoConfigurationError) do
|
85
|
-
propono_config.secret_key
|
86
|
-
end
|
87
|
-
end
|
88
|
-
|
89
|
-
def test_missing_queue_region_throws_exception
|
90
|
-
assert_raises(ProponoConfigurationError) do
|
91
|
-
propono_config.queue_region
|
92
|
-
end
|
93
|
-
end
|
94
|
-
|
95
80
|
def test_missing_application_name_throws_exception
|
96
81
|
assert_raises(ProponoConfigurationError) do
|
97
82
|
propono_config.application_name
|
@@ -6,9 +6,7 @@ module Propono
|
|
6
6
|
def propono_client
|
7
7
|
config_file = YAML.load_file( File.expand_path('../../config.yml', __FILE__))
|
8
8
|
@propono_client ||= Propono::Client.new do |config|
|
9
|
-
config.
|
10
|
-
config.secret_key = config_file['secret_key']
|
11
|
-
config.queue_region = config_file['queue_region']
|
9
|
+
config.aws_options = config_file['aws_options']
|
12
10
|
config.application_name = config_file['application_name']
|
13
11
|
end
|
14
12
|
end
|
@@ -58,6 +58,7 @@ module Propono
|
|
58
58
|
end
|
59
59
|
end
|
60
60
|
|
61
|
+
# Keep this test in sync with the one below, just with the config enabled
|
61
62
|
def test_drain_should_continue_if_queue_empty
|
62
63
|
@listener.expects(:read_messages_from_queue).with(@slow_queue, 10, long_poll: false).returns(false)
|
63
64
|
@listener.expects(:read_messages_from_queue).with(@queue, 10, long_poll: false).returns(false)
|
@@ -65,6 +66,16 @@ module Propono
|
|
65
66
|
assert true
|
66
67
|
end
|
67
68
|
|
69
|
+
# Keep this test in sync with the one above, just with the config disabled
|
70
|
+
def test_drain_ignores_slow_queue_if_disabled
|
71
|
+
propono_config.slow_queue_enabled = false
|
72
|
+
|
73
|
+
@listener.expects(:read_messages_from_queue).with(@slow_queue, 10, long_poll: false).never
|
74
|
+
@listener.expects(:read_messages_from_queue).with(@queue, 10, long_poll: false).returns(false)
|
75
|
+
@listener.drain
|
76
|
+
assert true
|
77
|
+
end
|
78
|
+
|
68
79
|
def test_drain_raises_with_nil_topic
|
69
80
|
listener = QueueListener.new(aws_client, propono_config, nil) {}
|
70
81
|
assert_raises ProponoError do
|
@@ -81,7 +92,7 @@ module Propono
|
|
81
92
|
|
82
93
|
def test_read_message_from_sqs
|
83
94
|
max_number_of_messages = 5
|
84
|
-
aws_client.expects(:read_from_sqs).with(@queue, max_number_of_messages, long_poll: true)
|
95
|
+
aws_client.expects(:read_from_sqs).with(@queue, max_number_of_messages, long_poll: true, visibility_timeout: nil)
|
85
96
|
@listener.send(:read_messages_from_queue, @queue, max_number_of_messages)
|
86
97
|
end
|
87
98
|
|
@@ -217,6 +228,7 @@ module Propono
|
|
217
228
|
@listener.send(:move_to_corrupt_queue, @sqs_message1)
|
218
229
|
end
|
219
230
|
|
231
|
+
# Keep this test in sync with the one below, just with the config enabled
|
220
232
|
def test_if_no_messages_read_from_normal_queue_read_from_slow_queue
|
221
233
|
main_queue = mock
|
222
234
|
@listener.stubs(main_queue: main_queue)
|
@@ -228,6 +240,20 @@ module Propono
|
|
228
240
|
@listener.send(:read_messages)
|
229
241
|
end
|
230
242
|
|
243
|
+
# Keep this test in sync with the one above, just with the config disabled
|
244
|
+
def ignore_slow_queue_if_disabled
|
245
|
+
propono_config.slow_queue_enabled = false
|
246
|
+
|
247
|
+
main_queue = mock
|
248
|
+
@listener.stubs(main_queue: main_queue)
|
249
|
+
slow_queue = mock
|
250
|
+
@listener.stubs(slow_queue: slow_queue)
|
251
|
+
|
252
|
+
@listener.expects(:read_messages_from_queue).with(main_queue, propono_config.num_messages_per_poll).returns(false)
|
253
|
+
@listener.expects(:read_messages_from_queue).with(slow_queue, 1).never
|
254
|
+
@listener.send(:read_messages)
|
255
|
+
end
|
256
|
+
|
231
257
|
def test_if_read_messages_from_normal_do_not_read_from_slow_queue
|
232
258
|
main_queue = mock
|
233
259
|
@listener.stubs(main_queue: main_queue)
|
data/test/test_helper.rb
CHANGED
@@ -15,9 +15,6 @@ class Minitest::Test
|
|
15
15
|
|
16
16
|
def propono_config
|
17
17
|
@propono_config ||= Propono::Configuration.new.tap do |c|
|
18
|
-
c.access_key = "test-access-key"
|
19
|
-
c.secret_key = "test-secret-key"
|
20
|
-
c.queue_region = "us-east-1"
|
21
18
|
c.application_name = "MyApp"
|
22
19
|
c.queue_suffix = ""
|
23
20
|
|
metadata
CHANGED
@@ -1,15 +1,17 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: propono
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 3.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
|
-
- MalcyL
|
8
7
|
- iHiD
|
9
|
-
|
8
|
+
- dougal
|
9
|
+
- ccare
|
10
|
+
- MalcyL
|
11
|
+
autorequire:
|
10
12
|
bindir: bin
|
11
13
|
cert_chain: []
|
12
|
-
date:
|
14
|
+
date: 2021-02-10 00:00:00.000000000 Z
|
13
15
|
dependencies:
|
14
16
|
- !ruby/object:Gem::Dependency
|
15
17
|
name: aws-sdk-sns
|
@@ -45,14 +47,14 @@ dependencies:
|
|
45
47
|
requirements:
|
46
48
|
- - "~>"
|
47
49
|
- !ruby/object:Gem::Version
|
48
|
-
version: '1
|
50
|
+
version: '2.1'
|
49
51
|
type: :development
|
50
52
|
prerelease: false
|
51
53
|
version_requirements: !ruby/object:Gem::Requirement
|
52
54
|
requirements:
|
53
55
|
- - "~>"
|
54
56
|
- !ruby/object:Gem::Version
|
55
|
-
version: '1
|
57
|
+
version: '2.1'
|
56
58
|
- !ruby/object:Gem::Dependency
|
57
59
|
name: rake
|
58
60
|
requirement: !ruby/object:Gem::Requirement
|
@@ -112,20 +114,20 @@ dependencies:
|
|
112
114
|
description: Pub / Sub Library using Amazon Web Services
|
113
115
|
email:
|
114
116
|
- jez.walker@gmail.com
|
115
|
-
- c.p.care@gmail.com
|
116
|
-
- malcolm@landonsonline.me.uk
|
117
117
|
executables: []
|
118
118
|
extensions: []
|
119
119
|
extra_rdoc_files: []
|
120
120
|
files:
|
121
|
+
- ".github/workflows/tests.yml"
|
121
122
|
- ".gitignore"
|
122
|
-
- ".
|
123
|
+
- ".ruby-version"
|
123
124
|
- CHANGELOG.md
|
124
125
|
- CONTRIBUTING.md
|
125
126
|
- Gemfile
|
126
127
|
- LICENCE.md
|
127
128
|
- README.md
|
128
129
|
- Rakefile
|
130
|
+
- docs/upgrading.md
|
129
131
|
- lib/propono.rb
|
130
132
|
- lib/propono/components/aws_client.rb
|
131
133
|
- lib/propono/components/aws_config.rb
|
@@ -158,11 +160,11 @@ files:
|
|
158
160
|
- test/services/queue_listener_test.rb
|
159
161
|
- test/test_helper.rb
|
160
162
|
- test/utils/hash_test.rb
|
161
|
-
homepage:
|
163
|
+
homepage: https://github.com/iHiD/propono/
|
162
164
|
licenses:
|
163
165
|
- MIT
|
164
166
|
metadata: {}
|
165
|
-
post_install_message:
|
167
|
+
post_install_message:
|
166
168
|
rdoc_options: []
|
167
169
|
require_paths:
|
168
170
|
- lib
|
@@ -173,13 +175,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
173
175
|
version: '0'
|
174
176
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
175
177
|
requirements:
|
176
|
-
- - "
|
178
|
+
- - ">="
|
177
179
|
- !ruby/object:Gem::Version
|
178
|
-
version:
|
180
|
+
version: '0'
|
179
181
|
requirements: []
|
180
|
-
|
181
|
-
|
182
|
-
signing_key:
|
182
|
+
rubygems_version: 3.0.3
|
183
|
+
signing_key:
|
183
184
|
specification_version: 4
|
184
185
|
summary: General purpose pub/sub library built on top of AWS SNS and SQS
|
185
186
|
test_files:
|
@@ -199,4 +200,3 @@ test_files:
|
|
199
200
|
- test/services/queue_listener_test.rb
|
200
201
|
- test/test_helper.rb
|
201
202
|
- test/utils/hash_test.rb
|
202
|
-
has_rdoc:
|