stockpile_cache 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -0
- data/Gemfile.lock +3 -3
- data/README.md +62 -2
- data/lib/stockpile.rb +33 -14
- data/lib/stockpile/cache.rb +8 -8
- data/lib/stockpile/cached_value_expirer.rb +2 -2
- data/lib/stockpile/cached_value_reader.rb +3 -3
- data/lib/stockpile/configuration.rb +12 -8
- data/lib/stockpile/constants.rb +1 -1
- data/lib/stockpile/{redis_connection.rb → default_redis_configuration.rb} +15 -10
- data/lib/stockpile/executor.rb +8 -6
- data/lib/stockpile/lock.rb +8 -7
- data/lib/stockpile/locked_execution_result.rb +4 -3
- data/lib/stockpile/redis_connections.rb +31 -0
- data/lib/stockpile/redis_connections_factory.rb +56 -0
- data/lib/stockpile/yaml_redis_configuration.rb +61 -0
- metadata +6 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e0cf5948f44622314a97af71ab6ce39aa3f59931092b7ac6d80f424e052be90d
|
|
4
|
+
data.tar.gz: bec173c3c11807350c3b4d4cb05f638fc4a90c71837882f7b64931db1dce8416
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: '08218ce4ab3946ae6b418dd0d72c354170415eff8b041d0e02bbc261ffca83fe4afd7718d9bb7907d2b80093afaeeaa8f83dc57e7b9a7aa4d91c1384ccf965b7'
|
|
7
|
+
data.tar.gz: fddd745d31dae1f5509cd3988e216d2fdfe5e520a971e58ef3ede3ed864c5b2e34191abef7b67bfab0115134fa4cdceec43fd404d8d8430c53df931b4f01bde1
|
data/CHANGELOG.md
CHANGED
data/Gemfile.lock
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
PATH
|
|
2
2
|
remote: .
|
|
3
3
|
specs:
|
|
4
|
-
stockpile_cache (1.0
|
|
4
|
+
stockpile_cache (1.2.0)
|
|
5
5
|
connection_pool
|
|
6
6
|
oj
|
|
7
7
|
rake
|
|
@@ -14,12 +14,12 @@ GEM
|
|
|
14
14
|
connection_pool (2.2.2)
|
|
15
15
|
diff-lcs (1.3)
|
|
16
16
|
jaro_winkler (1.5.3)
|
|
17
|
-
oj (3.9.
|
|
17
|
+
oj (3.9.2)
|
|
18
18
|
parallel (1.17.0)
|
|
19
19
|
parser (2.6.4.1)
|
|
20
20
|
ast (~> 2.4.0)
|
|
21
21
|
rainbow (3.0.0)
|
|
22
|
-
rake (13.0.
|
|
22
|
+
rake (13.0.1)
|
|
23
23
|
redis (4.1.3)
|
|
24
24
|
rspec (3.8.0)
|
|
25
25
|
rspec-core (~> 3.8.0)
|
data/README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Stockpile [![Build Status][ci-image]][ci] [![Code Climate][codeclimate-image]][codeclimate] [![Gem Version][version-image]][version]
|
|
2
2
|
Stockpile is a simple cache written in Ruby backed by Redis. It has built in
|
|
3
3
|
[cache-stampede](https://en.wikipedia.org/wiki/Cache_stampede) (also known as
|
|
4
|
-
dog-piling) protection.
|
|
4
|
+
dog-piling) protection and support for multiple Redis servers.
|
|
5
5
|
|
|
6
6
|
Can be used with any Ruby or Ruby on Rails project. Can be used as a replacement for
|
|
7
7
|
existing Ruby on Rails cache.
|
|
@@ -71,6 +71,7 @@ Following settings are supported:
|
|
|
71
71
|
| `STOCKPILE_REDIS_URL` | `redis_url` | URL of your Redis server that will be used for caching. Defaults to `redis://localhost:6379/1`. |
|
|
72
72
|
| `STOCKPILE_REDIS_SENTINELS` | `sentinels` | (optional) Comma separated list of Sentinels IPs for Redis. Defaults to `nil`. Example value: `8.8.8.8:42,8.8.4.4:42`. |
|
|
73
73
|
| `STOCKPILE_SLUMBER` | `slumber` | Timeout (in seconds) for stampede protection lock. After timeout passed in code will be executed instead of reading a value from cache. Defaults to `2`. |
|
|
74
|
+
| `STOCKPILE_CONFIGURATION_FILE` | `configuration_file` | (optional) `.yml` configuration file to read connection information from. See [Multiple Database](#multiple-database). |
|
|
74
75
|
|
|
75
76
|
## Usage
|
|
76
77
|
To use simply wrap your code into `perform_cached` block:
|
|
@@ -81,14 +82,73 @@ Stockpile.perform_cached(key: 'meaning_of_life', ttl: 42) do
|
|
|
81
82
|
end
|
|
82
83
|
```
|
|
83
84
|
|
|
84
|
-
`perform` method accepts
|
|
85
|
+
`perform` method accepts 4 named arguments:
|
|
85
86
|
|
|
86
87
|
| Argument | Meaning |
|
|
87
88
|
| ------------- | ------------- |
|
|
88
89
|
| `key` | Pointer in cache by which a value will be either looked up or stored in cache once code provided in block is executed. |
|
|
89
90
|
| `ttl` | (optional) Time in seconds for which a cached value will be stored. Defaults to 300 seconds (5 minutes). |
|
|
91
|
+
| `db` | (optional) Name of the Redis database to cache value in. Defaults to `:default` |
|
|
90
92
|
| `&block` | Block of code to execute; it's return value will be stored in cache. |
|
|
91
93
|
|
|
94
|
+
### Multiple Database
|
|
95
|
+
Stockpile comes with a support for multiple databases. A word of caution: unless
|
|
96
|
+
you have very good reason to run multiple databases within single instance of
|
|
97
|
+
Redis server you probably should avoid doing so as you will not see any performance
|
|
98
|
+
improvements in doing so.
|
|
99
|
+
|
|
100
|
+
To allow multi-database support you have to do two things. First you have to set
|
|
101
|
+
`configuration_file` setting to point at `.yml` containing your configuration.
|
|
102
|
+
You can do so by either setting a `STOCKPILE_CONFIGURATION_FILE` environment
|
|
103
|
+
variable or by executing a configuration block during runtime (for Rails create
|
|
104
|
+
`config/initializers/stockpile.rb` with following content):
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
Stockpile.configure do |configuration|
|
|
108
|
+
configuration.configuration_file = <PATH/TO/FILE>
|
|
109
|
+
end
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Second thing to do is to create a `.yml` configuration file. It has to have at
|
|
113
|
+
least one database definition. Providing `sentinels` is optional. Everything
|
|
114
|
+
else is mandatory:
|
|
115
|
+
|
|
116
|
+
```
|
|
117
|
+
---
|
|
118
|
+
master:
|
|
119
|
+
url: 'redis://redis-1-host:6379/1'
|
|
120
|
+
sentinels: '8.8.8.8:42,8.8.4.4:42'
|
|
121
|
+
pool_options:
|
|
122
|
+
size: 5
|
|
123
|
+
timeout: 5
|
|
124
|
+
|
|
125
|
+
commander:
|
|
126
|
+
url: 'redis://redis-2-host:6379/1'
|
|
127
|
+
pool_options:
|
|
128
|
+
size: 5
|
|
129
|
+
timeout: 5
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
To query different databases provide a corresponding `db:` param with
|
|
133
|
+
`perform_cached` method:
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
Stockpile.perform_cached(db: :master, key: 'meaning_of_life', ttl: 42) do
|
|
137
|
+
21 + 21
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
Stockpile.perform_cached(db: :commander, key: 'meaning_of_life', ttl: 21) do
|
|
141
|
+
21
|
|
142
|
+
end
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
If you do not provide a `db:` param then a `:default` database will be used; if
|
|
146
|
+
you do not define it in a configuration file your request will error out.
|
|
147
|
+
|
|
148
|
+
Using `configuration_file` setting will make Stockpile ignore all other
|
|
149
|
+
Redis connection related settings and it will read configuration from `.yml`
|
|
150
|
+
file instead.
|
|
151
|
+
|
|
92
152
|
## Caveats
|
|
93
153
|
There is no timeout or rescue set for code you will be running through the cache. If
|
|
94
154
|
you need to do either you have to handle it outside of Stockpile.
|
data/lib/stockpile.rb
CHANGED
|
@@ -18,10 +18,14 @@ require 'connection_pool'
|
|
|
18
18
|
require 'oj'
|
|
19
19
|
require 'redis'
|
|
20
20
|
require 'timeout'
|
|
21
|
+
require 'yaml'
|
|
21
22
|
|
|
22
23
|
require 'stockpile/constants'
|
|
23
24
|
require 'stockpile/configuration'
|
|
24
|
-
require 'stockpile/
|
|
25
|
+
require 'stockpile/redis_connections_factory'
|
|
26
|
+
require 'stockpile/default_redis_configuration'
|
|
27
|
+
require 'stockpile/yaml_redis_configuration'
|
|
28
|
+
require 'stockpile/redis_connections'
|
|
25
29
|
|
|
26
30
|
require 'stockpile/lock'
|
|
27
31
|
require 'stockpile/locked_execution_result'
|
|
@@ -36,16 +40,17 @@ require 'stockpile/executor'
|
|
|
36
40
|
# = Stockpile
|
|
37
41
|
#
|
|
38
42
|
# Simple cache with Redis as a backend and a built in cache-stampede
|
|
39
|
-
# protection. For more information on
|
|
40
|
-
# README.md file.
|
|
43
|
+
# protection and multiple Redis database support. For more information on
|
|
44
|
+
# general usage consider consulting README.md file.
|
|
41
45
|
#
|
|
42
46
|
# While interacting with the cache from within your application
|
|
43
47
|
# avoid re-using anything after :: notation as it is part of internal API
|
|
44
48
|
# and is subject to an un-announced breaking change.
|
|
45
49
|
#
|
|
46
|
-
# Stockpile provides
|
|
50
|
+
# Stockpile provides 6 methods as part of it's public API:
|
|
47
51
|
# * configuration
|
|
48
52
|
# * configure
|
|
53
|
+
# * expire_cached
|
|
49
54
|
# * perform_cached
|
|
50
55
|
# * redis
|
|
51
56
|
# * redis_connection_pool
|
|
@@ -59,7 +64,9 @@ module Stockpile
|
|
|
59
64
|
@configuration ||= Configuration.new
|
|
60
65
|
end
|
|
61
66
|
|
|
62
|
-
# API to configure cache dynamically during runtime.
|
|
67
|
+
# API to configure cache dynamically during runtime. Running dynamic
|
|
68
|
+
# configuration will rebuild connection pools releasing existing
|
|
69
|
+
# connections.
|
|
63
70
|
#
|
|
64
71
|
# @yield [configuration] Takes in a block of code of code that is setting
|
|
65
72
|
# or changing configuration values
|
|
@@ -70,17 +77,21 @@ module Stockpile
|
|
|
70
77
|
# @return [void]
|
|
71
78
|
def configure
|
|
72
79
|
yield(configuration)
|
|
80
|
+
@redis_connections = Stockpile::RedisConnectionsFactory.build_connections
|
|
81
|
+
|
|
73
82
|
nil
|
|
74
83
|
end
|
|
75
84
|
|
|
76
85
|
# Immediatelly expires a cached value for a given key.
|
|
77
86
|
#
|
|
78
87
|
# @params key [String] Key to expire
|
|
88
|
+
# @param db [Symbol] (optional) Which Redis database to expire data from.
|
|
89
|
+
# Defaults to `:default`
|
|
79
90
|
#
|
|
80
91
|
# @return [true, false] Returns true if value existed in cache and was
|
|
81
92
|
# succesfully expired. Returns false if value did not exist in cache.
|
|
82
|
-
def expire_cached(key:)
|
|
83
|
-
Stockpile::CachedValueExpirer.expire_cached(key: key)
|
|
93
|
+
def expire_cached(db: :default, key:)
|
|
94
|
+
Stockpile::CachedValueExpirer.expire_cached(db: db, key: key)
|
|
84
95
|
end
|
|
85
96
|
|
|
86
97
|
# Attempts to fetch a value from cache (for a given key). In case of miss
|
|
@@ -89,6 +100,8 @@ module Stockpile
|
|
|
89
100
|
#
|
|
90
101
|
# @param key [String] Key to use for a value lookup from cache or key
|
|
91
102
|
# to store value at once it is computed
|
|
103
|
+
# @param db [Symbol] (optional) Which Redis database to cache data in.
|
|
104
|
+
# Defaults to `:default`
|
|
92
105
|
# @param ttl [Integer] (optional) Time in seconds to expire cache after.
|
|
93
106
|
# Defaults to Stockpile::DEFAULT_TTL
|
|
94
107
|
#
|
|
@@ -98,8 +111,13 @@ module Stockpile
|
|
|
98
111
|
# Stockpile.perform_cached(key: 'meaning_of_life', ttl: 42) { 21 * 2 }
|
|
99
112
|
#
|
|
100
113
|
# @return Returns a result of block execution
|
|
101
|
-
def perform_cached(key:, ttl: Stockpile::DEFAULT_TTL, &block)
|
|
102
|
-
Stockpile::CachedValueReader.read_or_yield(
|
|
114
|
+
def perform_cached(db: :default, key:, ttl: Stockpile::DEFAULT_TTL, &block)
|
|
115
|
+
Stockpile::CachedValueReader.read_or_yield(
|
|
116
|
+
db: db,
|
|
117
|
+
key: key,
|
|
118
|
+
ttl: ttl,
|
|
119
|
+
&block
|
|
120
|
+
)
|
|
103
121
|
end
|
|
104
122
|
|
|
105
123
|
# API to communicate with Redis database backing cache up.
|
|
@@ -110,8 +128,8 @@ module Stockpile
|
|
|
110
128
|
# Store.redis { |r| r.set('meaning_of_life', 42) }
|
|
111
129
|
#
|
|
112
130
|
# @return Returns a result of interaction with Redis
|
|
113
|
-
def redis
|
|
114
|
-
|
|
131
|
+
def redis(db: :default)
|
|
132
|
+
redis_connections.with(db: db) do |connection|
|
|
115
133
|
yield connection
|
|
116
134
|
end
|
|
117
135
|
end
|
|
@@ -119,8 +137,9 @@ module Stockpile
|
|
|
119
137
|
# Accessor to connection pool. Defined on top level so it can be memoized
|
|
120
138
|
# on the topmost level
|
|
121
139
|
#
|
|
122
|
-
# @return [
|
|
123
|
-
|
|
124
|
-
|
|
140
|
+
# @return [Stockpile::RedisConnections] RedisConnections object holding all defined
|
|
141
|
+
# connection pools
|
|
142
|
+
def redis_connections
|
|
143
|
+
@redis_connections ||= Stockpile::RedisConnectionsFactory.build_connections
|
|
125
144
|
end
|
|
126
145
|
end
|
data/lib/stockpile/cache.rb
CHANGED
|
@@ -22,21 +22,21 @@ module Stockpile
|
|
|
22
22
|
module Cache
|
|
23
23
|
module_function
|
|
24
24
|
|
|
25
|
-
def get(key:)
|
|
26
|
-
value_from_cache = Stockpile.redis { |r| r.get(key) }
|
|
25
|
+
def get(db: :default, key:)
|
|
26
|
+
value_from_cache = Stockpile.redis(db: db) { |r| r.get(key) }
|
|
27
27
|
Oj.load(value_from_cache) if value_from_cache
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
-
def get_deferred(key:)
|
|
31
|
-
sleep(Stockpile::SLUMBER_COOLDOWN) until Stockpile.redis { |r| r.exists(key) }
|
|
32
|
-
value_from_cache = Stockpile.redis { |r| r.get(key) }
|
|
30
|
+
def get_deferred(db: :default, key:)
|
|
31
|
+
sleep(Stockpile::SLUMBER_COOLDOWN) until Stockpile.redis(db: db) { |r| r.exists(key) }
|
|
32
|
+
value_from_cache = Stockpile.redis(db: db) { |r| r.get(key) }
|
|
33
33
|
Oj.load(value_from_cache)
|
|
34
34
|
end
|
|
35
35
|
|
|
36
|
-
def set(key:, payload:, ttl:)
|
|
36
|
+
def set(db: :default, key:, payload:, ttl:)
|
|
37
37
|
payload = Oj.dump(payload)
|
|
38
|
-
Stockpile.redis { |r| r.set(key, payload) }
|
|
39
|
-
Stockpile.redis { |r| r.expire(key, ttl) }
|
|
38
|
+
Stockpile.redis(db: db) { |r| r.set(key, payload) }
|
|
39
|
+
Stockpile.redis(db: db) { |r| r.expire(key, ttl) }
|
|
40
40
|
end
|
|
41
41
|
end
|
|
42
42
|
end
|
|
@@ -22,11 +22,11 @@ module Stockpile
|
|
|
22
22
|
module CachedValueReader
|
|
23
23
|
module_function
|
|
24
24
|
|
|
25
|
-
def read_or_yield(key:, ttl:, &block)
|
|
26
|
-
if (result = Stockpile::Cache.get(key: key))
|
|
25
|
+
def read_or_yield(db: :default, key:, ttl:, &block)
|
|
26
|
+
if (result = Stockpile::Cache.get(db: db, key: key))
|
|
27
27
|
result
|
|
28
28
|
else
|
|
29
|
-
Stockpile::Executor.perform(key: key, ttl: ttl, &block)
|
|
29
|
+
Stockpile::Executor.perform(db: db, key: key, ttl: ttl, &block)
|
|
30
30
|
end
|
|
31
31
|
end
|
|
32
32
|
end
|
|
@@ -20,20 +20,25 @@ module Stockpile
|
|
|
20
20
|
# Holds configuration for cache with writeable attributes allowing
|
|
21
21
|
# dynamic change of configuration during runtime
|
|
22
22
|
class Configuration
|
|
23
|
-
attr_accessor :
|
|
24
|
-
:redis_url, :sentinels, :slumber
|
|
23
|
+
attr_accessor :configuration_file, :connection_pool, :connection_timeout,
|
|
24
|
+
:lock_expiration, :redis_url, :sentinels, :slumber
|
|
25
25
|
|
|
26
26
|
def initialize
|
|
27
|
+
@configuration_file = extract_configuration_file
|
|
27
28
|
@connection_pool = extract_connection_pool
|
|
28
29
|
@connection_timeout = extract_connection_timeout
|
|
29
30
|
@lock_expiration = extract_lock_expiration
|
|
30
31
|
@redis_url = extract_redis_url
|
|
31
|
-
@sentinels =
|
|
32
|
+
@sentinels = extract_sentinels
|
|
32
33
|
@slumber = extract_slumber
|
|
33
34
|
end
|
|
34
35
|
|
|
35
36
|
private
|
|
36
37
|
|
|
38
|
+
def extract_configuration_file
|
|
39
|
+
ENV.fetch('STOCKPILE_CONFIGURATION_FILE', nil)
|
|
40
|
+
end
|
|
41
|
+
|
|
37
42
|
def extract_connection_pool
|
|
38
43
|
ENV.fetch(
|
|
39
44
|
'STOCKPILE_CONNECTION_POOL',
|
|
@@ -69,11 +74,10 @@ module Stockpile
|
|
|
69
74
|
).to_i
|
|
70
75
|
end
|
|
71
76
|
|
|
72
|
-
def
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
end
|
|
77
|
+
def extract_sentinels
|
|
78
|
+
Stockpile::RedisConnectionsFactory.process_sentinels(
|
|
79
|
+
sentinels: ENV.fetch('STOCKPILE_REDIS_SENTINELS', '')
|
|
80
|
+
)
|
|
77
81
|
end
|
|
78
82
|
end
|
|
79
83
|
end
|
data/lib/stockpile/constants.rb
CHANGED
|
@@ -15,27 +15,32 @@
|
|
|
15
15
|
# limitations under the License.
|
|
16
16
|
|
|
17
17
|
module Stockpile
|
|
18
|
-
# == Stockpile::
|
|
18
|
+
# == Stockpile::DefaultRedisConfiguration
|
|
19
19
|
#
|
|
20
|
-
#
|
|
21
|
-
#
|
|
22
|
-
|
|
20
|
+
# Confiuration object for a single Redis database cache setup.
|
|
21
|
+
# Reads values out of environment, default values or uses
|
|
22
|
+
# configuration provided during runtime.
|
|
23
|
+
module DefaultRedisConfiguration
|
|
23
24
|
module_function
|
|
24
25
|
|
|
25
|
-
def
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
26
|
+
def configuration
|
|
27
|
+
[
|
|
28
|
+
{
|
|
29
|
+
db: :default,
|
|
30
|
+
pool_configuration: pool_configuration,
|
|
31
|
+
redis_configuration: redis_configuration
|
|
32
|
+
}
|
|
33
|
+
]
|
|
29
34
|
end
|
|
30
35
|
|
|
31
|
-
def
|
|
36
|
+
def redis_configuration
|
|
32
37
|
{
|
|
33
38
|
url: redis_url,
|
|
34
39
|
sentinels: sentinels
|
|
35
40
|
}.delete_if { |_k, v| v.nil? || v.empty? }
|
|
36
41
|
end
|
|
37
42
|
|
|
38
|
-
def
|
|
43
|
+
def pool_configuration
|
|
39
44
|
{
|
|
40
45
|
size: pool_size,
|
|
41
46
|
timeout: connection_timeout
|
data/lib/stockpile/executor.rb
CHANGED
|
@@ -22,13 +22,14 @@ module Stockpile
|
|
|
22
22
|
# value to appear in cache instead. Will timeout after given amount of time
|
|
23
23
|
# and will execute block if no value can be read from cache.
|
|
24
24
|
class Executor
|
|
25
|
-
attr_reader :key, :ttl
|
|
25
|
+
attr_reader :db, :key, :ttl
|
|
26
26
|
|
|
27
|
-
def self.perform(key:, ttl:, &block)
|
|
28
|
-
new(key, ttl).perform(&block)
|
|
27
|
+
def self.perform(db: :default, key:, ttl:, &block)
|
|
28
|
+
new(db, key, ttl).perform(&block)
|
|
29
29
|
end
|
|
30
30
|
|
|
31
|
-
def initialize(key, ttl)
|
|
31
|
+
def initialize(db, key, ttl)
|
|
32
|
+
@db = db
|
|
32
33
|
@key = key
|
|
33
34
|
@ttl = ttl
|
|
34
35
|
end
|
|
@@ -44,13 +45,14 @@ module Stockpile
|
|
|
44
45
|
private
|
|
45
46
|
|
|
46
47
|
def execution
|
|
47
|
-
@execution ||= Stockpile::Lock.perform_locked(lock_key: lock_key) do
|
|
48
|
+
@execution ||= Stockpile::Lock.perform_locked(db: db, lock_key: lock_key) do
|
|
48
49
|
yield
|
|
49
50
|
end
|
|
50
51
|
end
|
|
51
52
|
|
|
52
53
|
def cache_and_release_execution
|
|
53
54
|
Stockpile::Cache.set(
|
|
55
|
+
db: db,
|
|
54
56
|
key: key,
|
|
55
57
|
payload: execution.result,
|
|
56
58
|
ttl: ttl
|
|
@@ -66,7 +68,7 @@ module Stockpile
|
|
|
66
68
|
|
|
67
69
|
def wait_for_cache_or_yield
|
|
68
70
|
Timeout.timeout(Stockpile.configuration.slumber) do
|
|
69
|
-
Stockpile::Cache.get_deferred(key: key)
|
|
71
|
+
Stockpile::Cache.get_deferred(db: db, key: key)
|
|
70
72
|
end
|
|
71
73
|
rescue Timeout::Error
|
|
72
74
|
yield
|
data/lib/stockpile/lock.rb
CHANGED
|
@@ -23,13 +23,14 @@ module Stockpile
|
|
|
23
23
|
# Stockpile::LockedExcutionResult will hold Stockpile::FailedLockExecution
|
|
24
24
|
# as a result of execution
|
|
25
25
|
class Lock
|
|
26
|
-
attr_reader :lock_key
|
|
26
|
+
attr_reader :db, :lock_key
|
|
27
27
|
|
|
28
|
-
def self.perform_locked(lock_key:, &block)
|
|
29
|
-
new(lock_key).perform_locked(&block)
|
|
28
|
+
def self.perform_locked(db: :default, lock_key:, &block)
|
|
29
|
+
new(db, lock_key).perform_locked(&block)
|
|
30
30
|
end
|
|
31
31
|
|
|
32
|
-
def initialize(lock_key)
|
|
32
|
+
def initialize(db, lock_key)
|
|
33
|
+
@db = db
|
|
33
34
|
@lock_key = lock_key
|
|
34
35
|
end
|
|
35
36
|
|
|
@@ -44,7 +45,7 @@ module Stockpile
|
|
|
44
45
|
private
|
|
45
46
|
|
|
46
47
|
def failed_execution
|
|
47
|
-
Stockpile::LockedExcutionResult.new(result: failed_lock, lock_key: lock_key)
|
|
48
|
+
Stockpile::LockedExcutionResult.new(db: db, result: failed_lock, lock_key: lock_key)
|
|
48
49
|
end
|
|
49
50
|
|
|
50
51
|
def failed_lock
|
|
@@ -52,11 +53,11 @@ module Stockpile
|
|
|
52
53
|
end
|
|
53
54
|
|
|
54
55
|
def lock
|
|
55
|
-
Stockpile.redis { |r| r.set(lock_key, 1, nx: true, ex: Stockpile.configuration.lock_expiration) }
|
|
56
|
+
Stockpile.redis(db: db) { |r| r.set(lock_key, 1, nx: true, ex: Stockpile.configuration.lock_expiration) }
|
|
56
57
|
end
|
|
57
58
|
|
|
58
59
|
def successful_execution
|
|
59
|
-
Stockpile::LockedExcutionResult.new(result: yield, lock_key: lock_key)
|
|
60
|
+
Stockpile::LockedExcutionResult.new(db: db, result: yield, lock_key: lock_key)
|
|
60
61
|
end
|
|
61
62
|
end
|
|
62
63
|
end
|
|
@@ -19,15 +19,16 @@ module Stockpile
|
|
|
19
19
|
#
|
|
20
20
|
# Wrapper containing result of locked execution
|
|
21
21
|
class LockedExcutionResult
|
|
22
|
-
attr_reader :lock_key, :result
|
|
22
|
+
attr_reader :db, :lock_key, :result
|
|
23
23
|
|
|
24
|
-
def initialize(lock_key:, result:)
|
|
24
|
+
def initialize(db: :default, lock_key:, result:)
|
|
25
|
+
@db = db
|
|
25
26
|
@lock_key = lock_key
|
|
26
27
|
@result = result
|
|
27
28
|
end
|
|
28
29
|
|
|
29
30
|
def release_lock
|
|
30
|
-
Stockpile.redis { |r| r.expire(lock_key, 0) }
|
|
31
|
+
Stockpile.redis(db: db) { |r| r.expire(lock_key, 0) }
|
|
31
32
|
end
|
|
32
33
|
|
|
33
34
|
def success?
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 ConvertKit, LLC
|
|
4
|
+
#
|
|
5
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
6
|
+
# you may not use this file except in compliance with the License.
|
|
7
|
+
# You may obtain a copy of the License at
|
|
8
|
+
#
|
|
9
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
|
10
|
+
#
|
|
11
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
12
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
13
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
14
|
+
# See the License for the specific language governing permissions and
|
|
15
|
+
# limitations under the License.
|
|
16
|
+
|
|
17
|
+
module Stockpile
|
|
18
|
+
# == Stockpile::RedisConnections
|
|
19
|
+
#
|
|
20
|
+
# Wrapper around pools of Redis connections to allow multiple
|
|
21
|
+
# Redis database support
|
|
22
|
+
module RedisConnections
|
|
23
|
+
module_function
|
|
24
|
+
|
|
25
|
+
def with(db:)
|
|
26
|
+
instance_variable_get("@#{db}".to_sym).with do |connection|
|
|
27
|
+
yield connection
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 ConvertKit, LLC
|
|
4
|
+
#
|
|
5
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
6
|
+
# you may not use this file except in compliance with the License.
|
|
7
|
+
# You may obtain a copy of the License at
|
|
8
|
+
#
|
|
9
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
|
10
|
+
#
|
|
11
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
12
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
13
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
14
|
+
# See the License for the specific language governing permissions and
|
|
15
|
+
# limitations under the License.
|
|
16
|
+
|
|
17
|
+
module Stockpile
|
|
18
|
+
# == Stockpile::RedisConnectionsFactory
|
|
19
|
+
#
|
|
20
|
+
# Builds out connection pools out of provided configuration. Configurations
|
|
21
|
+
# are built with `*RedisConfiguration` classes. Providing a `.yml` file will
|
|
22
|
+
# override everything else and use that to build a config.
|
|
23
|
+
module RedisConnectionsFactory
|
|
24
|
+
module_function
|
|
25
|
+
|
|
26
|
+
def build_connections
|
|
27
|
+
configuration.each do |database|
|
|
28
|
+
pool = ConnectionPool.new(database[:pool_configuration]) do
|
|
29
|
+
Redis.new(database[:redis_configuration])
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
RedisConnections.instance_variable_set(
|
|
33
|
+
"@#{database[:db]}".to_sym,
|
|
34
|
+
pool
|
|
35
|
+
)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
RedisConnections
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def configuration
|
|
42
|
+
if Stockpile.configuration.configuration_file
|
|
43
|
+
Stockpile::YamlRedisConfiguration.configuration
|
|
44
|
+
else
|
|
45
|
+
Stockpile::DefaultRedisConfiguration.configuration
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def process_sentinels(sentinels:)
|
|
50
|
+
sentinels.split(',').map do |sentinel|
|
|
51
|
+
host, port = sentinel.split(':')
|
|
52
|
+
{ host: host, port: port.to_i }
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
end
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Copyright 2019 ConvertKit, LLC
|
|
4
|
+
#
|
|
5
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
6
|
+
# you may not use this file except in compliance with the License.
|
|
7
|
+
# You may obtain a copy of the License at
|
|
8
|
+
#
|
|
9
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
|
10
|
+
#
|
|
11
|
+
# Unless required by applicable law or agreed to in writing, software
|
|
12
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
13
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
14
|
+
# See the License for the specific language governing permissions and
|
|
15
|
+
# limitations under the License.
|
|
16
|
+
|
|
17
|
+
module Stockpile
|
|
18
|
+
# == Stockpile::YamlRedisConfiguration
|
|
19
|
+
#
|
|
20
|
+
# Confiuration object a multiple Redis database cache setup. Reads
|
|
21
|
+
# configuration out of provided `.yml` file.
|
|
22
|
+
module YamlRedisConfiguration
|
|
23
|
+
module_function
|
|
24
|
+
|
|
25
|
+
def configuration
|
|
26
|
+
parsed_configuration.map do |database, settings|
|
|
27
|
+
{
|
|
28
|
+
db: database,
|
|
29
|
+
pool_configuration: extract_pool(settings: settings),
|
|
30
|
+
redis_configuration: extract_redis(settings: settings)
|
|
31
|
+
}
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def extract_redis(settings:)
|
|
36
|
+
sentinels = Stockpile::RedisConnectionsFactory.process_sentinels(
|
|
37
|
+
sentinels: settings['sentinels'] || ''
|
|
38
|
+
)
|
|
39
|
+
|
|
40
|
+
{
|
|
41
|
+
url: settings['url'],
|
|
42
|
+
sentinels: sentinels
|
|
43
|
+
}.delete_if { |_k, v| v.nil? || v.empty? }
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def extract_pool(settings:)
|
|
47
|
+
{
|
|
48
|
+
size: settings.dig('pool_options', 'size'),
|
|
49
|
+
timeout: settings.dig('pool_options', 'timeout')
|
|
50
|
+
}
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def parsed_configuration
|
|
54
|
+
YAML.safe_load(raw_configuration)
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def raw_configuration
|
|
58
|
+
File.open(Stockpile.configuration.configuration_file).read
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: stockpile_cache
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- ConvertKit, LLC
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2019-
|
|
11
|
+
date: 2019-11-28 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: connection_pool
|
|
@@ -93,11 +93,14 @@ files:
|
|
|
93
93
|
- lib/stockpile/cached_value_reader.rb
|
|
94
94
|
- lib/stockpile/configuration.rb
|
|
95
95
|
- lib/stockpile/constants.rb
|
|
96
|
+
- lib/stockpile/default_redis_configuration.rb
|
|
96
97
|
- lib/stockpile/executor.rb
|
|
97
98
|
- lib/stockpile/failed_lock_execution.rb
|
|
98
99
|
- lib/stockpile/lock.rb
|
|
99
100
|
- lib/stockpile/locked_execution_result.rb
|
|
100
|
-
- lib/stockpile/
|
|
101
|
+
- lib/stockpile/redis_connections.rb
|
|
102
|
+
- lib/stockpile/redis_connections_factory.rb
|
|
103
|
+
- lib/stockpile/yaml_redis_configuration.rb
|
|
101
104
|
- lib/stockpile_cache.rb
|
|
102
105
|
- stockpile-cache.gemspec
|
|
103
106
|
homepage:
|