stockpile_cache 1.0.0 → 1.3.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f846deab3025ca6771b8cf40a4ff55c358256164208d3dd9409667efc51aa013
4
- data.tar.gz: 0d9dbe79e95be12dbbb6db1edc7f0c3e2d9d40cc9731c61362d5f415a81aaec9
3
+ metadata.gz: 6ad32124765de7dec002510817eba80abf37eb7e42403de931e6c74cc9178842
4
+ data.tar.gz: b0242fe4109a69ce917997181ec76b00b47c0cb8c02794d88edf75276b5d62fc
5
5
  SHA512:
6
- metadata.gz: c7921d065ed50ff8fab43453a2d5532d724939e55dea2f98df06e8a92ef1bbfb7b513fb87ffcf5841ac584a4a7b789220ae06ffb8e30ae6d33ef4c2469457da5
7
- data.tar.gz: a61bc77c28a775e309b6f63f9af18891e54e0027e745ca58cb68fe98524cd931bc5fe1a7087bd479f8d8c1b9e557cc7797ccf309394415d83a25af589589d307
6
+ metadata.gz: 1540e626290d9209a0820c8a8f9830f9bc14cf8d10569bf6279b4369a0da35843d0a19ef99f90e0ab21de31c3653a24d5658285e410085aeda4e2d7209658cc6
7
+ data.tar.gz: 7bf019df445a2a1acd3a70d1400a65d06b09056a35ea287ea3b469fd7fb65753530d47cf2105a93f8f6b7dc1348409ba5ba5680464ce46b3d897fd55c7a00503
@@ -0,0 +1,23 @@
1
+ # Changelog
2
+
3
+ ## 1.3.1
4
+ - Parsing ERB in YAML configs
5
+ - Fill in missing README about cache expiration
6
+
7
+ ## 1.3.0
8
+ - Allowing optional compression of cached content
9
+
10
+ ## 1.2.0
11
+ - Adding support for multiple Redis databases/server
12
+
13
+ ## 1.1.0
14
+ - Allowing expiration of cached value
15
+
16
+ ## 1.0.1
17
+ - Adding CHANGELOG.md
18
+ - Removing homepage from gemspec as we don't have proper one yet
19
+ - Adding source code URL
20
+ - Allowing to connect to Redis with no sentinels
21
+
22
+ ## 1.0.0
23
+ - Initial commit!
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- stockpile_cache (1.0.0)
4
+ stockpile_cache (1.3.1)
5
5
  connection_pool
6
6
  oj
7
7
  rake
@@ -14,13 +14,13 @@ GEM
14
14
  connection_pool (2.2.2)
15
15
  diff-lcs (1.3)
16
16
  jaro_winkler (1.5.3)
17
- oj (3.9.1)
17
+ oj (3.10.6)
18
18
  parallel (1.17.0)
19
19
  parser (2.6.4.1)
20
20
  ast (~> 2.4.0)
21
21
  rainbow (3.0.0)
22
- rake (12.3.3)
23
- redis (4.1.2)
22
+ rake (13.0.1)
23
+ redis (4.1.4)
24
24
  rspec (3.8.0)
25
25
  rspec-core (~> 3.8.0)
26
26
  rspec-expectations (~> 3.8.0)
@@ -53,4 +53,4 @@ DEPENDENCIES
53
53
  stockpile_cache!
54
54
 
55
55
  BUNDLED WITH
56
- 2.0.2
56
+ 2.1.4
data/README.md CHANGED
@@ -1,7 +1,7 @@
1
- # Stockpile [![Build Status][ci-image]][ci] [![Code Climate][codeclimate-image]][codeclimate]
1
+ # Stockpile [![Build Status][ci-image]][ci] [![Code Climate][codeclimate-image]][codeclimate] [![Gem Version][version-image]][version]
2
2
  Stockpile is a simple cache written in Ruby backed by Redis. It has built in
3
3
  [cache-stampede](https://en.wikipedia.org/wiki/Cache_stampede) (also known as
4
- dog-piling) protection.
4
+ dog-piling) protection and support for multiple Redis servers.
5
5
 
6
6
  Can be used with any Ruby or Ruby on Rails project. Can be used as a replacement for
7
7
  existing Ruby on Rails cache.
@@ -71,6 +71,7 @@ Following settings are supported:
71
71
  | `STOCKPILE_REDIS_URL` | `redis_url` | URL of your Redis server that will be used for caching. Defaults to `redis://localhost:6379/1`. |
72
72
  | `STOCKPILE_REDIS_SENTINELS` | `sentinels` | (optional) Comma separated list of Sentinels IPs for Redis. Defaults to `nil`. Example value: `8.8.8.8:42,8.8.4.4:42`. |
73
73
  | `STOCKPILE_SLUMBER` | `slumber` | Timeout (in seconds) for stampede protection lock. After timeout passed in code will be executed instead of reading a value from cache. Defaults to `2`. |
74
+ | `STOCKPILE_CONFIGURATION_FILE` | `configuration_file` | (optional) `.yml` configuration file to read connection information from. See [Multiple Database](#multiple-database). |
74
75
 
75
76
  ## Usage
76
77
  To use simply wrap your code into `perform_cached` block:
@@ -81,14 +82,117 @@ Stockpile.perform_cached(key: 'meaning_of_life', ttl: 42) do
81
82
  end
82
83
  ```
83
84
 
84
- `perform` method accepts 3 named arguments:
85
+ `perform` method accepts 4 named arguments:
85
86
 
86
87
  | Argument | Meaning |
87
88
  | ------------- | ------------- |
88
89
  | `key` | Pointer in cache by which a value will be either looked up or stored in cache once code provided in block is executed. |
89
90
  | `ttl` | (optional) Time in seconds for which a cached value will be stored. Defaults to 300 seconds (5 minutes). |
91
+ | `db` | (optional) Name of the Redis database to cache value in. Defaults to `:default` |
90
92
  | `&block` | Block of code to execute; it's return value will be stored in cache. |
91
93
 
94
+ To expire your cache immediately run:
95
+
96
+ ```
97
+ Stockpile.expire_cached(key: 'meaning_of_life')
98
+ ```
99
+
100
+ ### Multiple Database
101
+ Stockpile comes with a support for multiple databases. A word of caution: unless
102
+ you have very good reason to run multiple databases within single instance of
103
+ Redis server you probably should avoid doing so as you will not see any performance
104
+ improvements in doing so.
105
+
106
+ To allow multi-database support you have to do two things. First you have to set
107
+ `configuration_file` setting to point at `.yml` containing your configuration.
108
+ You can do so by either setting a `STOCKPILE_CONFIGURATION_FILE` environment
109
+ variable or by executing a configuration block during runtime (for Rails create
110
+ `config/initializers/stockpile.rb` with following content):
111
+
112
+ ```
113
+ Stockpile.configure do |configuration|
114
+ configuration.configuration_file = <PATH/TO/FILE>
115
+ end
116
+ ```
117
+
118
+ Second thing to do is to create a `.yml` configuration file. It has to have at
119
+ least one database definition. Providing `sentinels` is optional. Everything
120
+ else is mandatory:
121
+
122
+ ```
123
+ ---
124
+ master:
125
+ url: 'redis://redis-1-host:6379/1'
126
+ sentinels: '8.8.8.8:42,8.8.4.4:42'
127
+ pool_options:
128
+ size: 5
129
+ timeout: 5
130
+
131
+ commander:
132
+ url: 'redis://redis-2-host:6379/1'
133
+ pool_options:
134
+ size: 5
135
+ timeout: 5
136
+ ```
137
+
138
+ To query different databases provide a corresponding `db:` param with
139
+ `perform_cached` method:
140
+
141
+ ```
142
+ Stockpile.perform_cached(db: :master, key: 'meaning_of_life', ttl: 42) do
143
+ 21 + 21
144
+ end
145
+
146
+ Stockpile.perform_cached(db: :commander, key: 'meaning_of_life', ttl: 21) do
147
+ 21
148
+ end
149
+ ```
150
+
151
+ If you do not provide a `db:` param then a `:default` database will be used; if
152
+ you do not define it in a configuration file your request will error out.
153
+
154
+ Using `configuration_file` setting will make Stockpile ignore all other
155
+ Redis connection related settings and it will read configuration from `.yml`
156
+ file instead.
157
+
158
+ ### Compression of Cached Content
159
+ Stockpile optionally supports compression of cached content; you will not see
160
+ much benefit from compressing small strings but once you start caching bigger
161
+ payloads like fragments of HTML you could see some improvements by using
162
+ compression. To use compression you will have to use configuration file set by
163
+ `STOCKPILE_CONFIGURATION_FILE`.
164
+
165
+ To enable compression you have to do two things. First you have to set
166
+ `configuration_file` setting to point at `.yml` containing your configuration.
167
+ You can do so by either setting a `STOCKPILE_CONFIGURATION_FILE` environment
168
+ variable or by executing a configuration block during runtime (for Rails create
169
+ `config/initializers/stockpile.rb` with following content):
170
+
171
+ ```
172
+ Stockpile.configure do |configuration|
173
+ configuration.configuration_file = <PATH/TO/FILE>
174
+ end
175
+ ```
176
+
177
+ Second thing to do is to create a `.yml` configuration file. It has to have at
178
+ least one database definition. Providing `sentinels` and `compression` is
179
+ optional. Everything else is mandatory:
180
+
181
+
182
+ ```
183
+ ---
184
+ master:
185
+ url: 'redis://redis-1-host:6379/1'
186
+ sentinels: '8.8.8.8:42,8.8.4.4:42'
187
+ compression: true
188
+ pool_options:
189
+ size: 5
190
+ timeout: 5
191
+ ```
192
+
193
+ From that point everything that will be cached in `master` database will be
194
+ compressed.
195
+
92
196
  ## Caveats
93
197
  There is no timeout or rescue set for code you will be running through the cache. If
94
198
  you need to do either you have to handle it outside of Stockpile.
@@ -133,3 +237,5 @@ conduct](https://github.com/ConvertKit/stockpile_cache/blob/master/CODE_OF_CONDU
133
237
  [ci-image]: https://circleci.com/gh/ConvertKit/stockpile_cache.svg?style=svg
134
238
  [codeclimate]: https://codeclimate.com/github/ConvertKit/stockpile_cache/maintainability
135
239
  [codeclimate-image]: https://api.codeclimate.com/v1/badges/f9ca3b6dda3b492b125e/maintainability
240
+ [version]: https://badge.fury.io/rb/stockpile_cache
241
+ [version-image]: https://badge.fury.io/rb/stockpile_cache.svg
@@ -14,14 +14,20 @@
14
14
  # See the License for the specific language governing permissions and
15
15
  # limitations under the License.
16
16
 
17
+ require 'base64'
17
18
  require 'connection_pool'
18
19
  require 'oj'
19
20
  require 'redis'
20
21
  require 'timeout'
22
+ require 'yaml'
23
+ require 'zlib'
21
24
 
22
25
  require 'stockpile/constants'
23
26
  require 'stockpile/configuration'
24
- require 'stockpile/redis_connection'
27
+ require 'stockpile/redis_connections_factory'
28
+ require 'stockpile/default_redis_configuration'
29
+ require 'stockpile/yaml_redis_configuration'
30
+ require 'stockpile/redis_connections'
25
31
 
26
32
  require 'stockpile/lock'
27
33
  require 'stockpile/locked_execution_result'
@@ -29,22 +35,24 @@ require 'stockpile/failed_lock_execution'
29
35
 
30
36
  require 'stockpile/cache'
31
37
  require 'stockpile/cached_value_reader'
38
+ require 'stockpile/cached_value_expirer'
32
39
 
33
40
  require 'stockpile/executor'
34
41
 
35
42
  # = Stockpile
36
43
  #
37
44
  # Simple cache with Redis as a backend and a built in cache-stampede
38
- # protection. For more information on general usage consider consulting
39
- # README.md file.
45
+ # protection and multiple Redis database support. For more information on
46
+ # general usage consider consulting README.md file.
40
47
  #
41
48
  # While interacting with the cache from within your application
42
49
  # avoid re-using anything after :: notation as it is part of internal API
43
50
  # and is subject to an un-announced breaking change.
44
51
  #
45
- # Stockpile provides 5 methods as part of it's public API:
52
+ # Stockpile provides 6 methods as part of it's public API:
46
53
  # * configuration
47
54
  # * configure
55
+ # * expire_cached
48
56
  # * perform_cached
49
57
  # * redis
50
58
  # * redis_connection_pool
@@ -58,7 +66,9 @@ module Stockpile
58
66
  @configuration ||= Configuration.new
59
67
  end
60
68
 
61
- # API to configure cache dynamically during runtime.
69
+ # API to configure cache dynamically during runtime. Running dynamic
70
+ # configuration will rebuild connection pools releasing existing
71
+ # connections.
62
72
  #
63
73
  # @yield [configuration] Takes in a block of code of code that is setting
64
74
  # or changing configuration values
@@ -69,15 +79,31 @@ module Stockpile
69
79
  # @return [void]
70
80
  def configure
71
81
  yield(configuration)
82
+ @redis_connections = Stockpile::RedisConnectionsFactory.build_connections
83
+
72
84
  nil
73
85
  end
74
86
 
87
+ # Immediatelly expires a cached value for a given key.
88
+ #
89
+ # @params key [String] Key to expire
90
+ # @param db [Symbol] (optional) Which Redis database to expire data from.
91
+ # Defaults to `:default`
92
+ #
93
+ # @return [true, false] Returns true if value existed in cache and was
94
+ # succesfully expired. Returns false if value did not exist in cache.
95
+ def expire_cached(db: :default, key:)
96
+ Stockpile::CachedValueExpirer.expire_cached(db: db, key: key)
97
+ end
98
+
75
99
  # Attempts to fetch a value from cache (for a given key). In case of miss
76
100
  # will execute given block of code and cache it's result at the provided
77
101
  # key for a specified TTL.
78
102
  #
79
103
  # @param key [String] Key to use for a value lookup from cache or key
80
104
  # to store value at once it is computed
105
+ # @param db [Symbol] (optional) Which Redis database to cache data in.
106
+ # Defaults to `:default`
81
107
  # @param ttl [Integer] (optional) Time in seconds to expire cache after.
82
108
  # Defaults to Stockpile::DEFAULT_TTL
83
109
  #
@@ -87,8 +113,13 @@ module Stockpile
87
113
  # Stockpile.perform_cached(key: 'meaning_of_life', ttl: 42) { 21 * 2 }
88
114
  #
89
115
  # @return Returns a result of block execution
90
- def perform_cached(key:, ttl: Stockpile::DEFAULT_TTL, &block)
91
- Stockpile::CachedValueReader.read_or_yield(key: key, ttl: ttl, &block)
116
+ def perform_cached(db: :default, key:, ttl: Stockpile::DEFAULT_TTL, &block)
117
+ Stockpile::CachedValueReader.read_or_yield(
118
+ db: db,
119
+ key: key,
120
+ ttl: ttl,
121
+ &block
122
+ )
92
123
  end
93
124
 
94
125
  # API to communicate with Redis database backing cache up.
@@ -99,8 +130,8 @@ module Stockpile
99
130
  # Store.redis { |r| r.set('meaning_of_life', 42) }
100
131
  #
101
132
  # @return Returns a result of interaction with Redis
102
- def redis
103
- redis_connection_pool.with do |connection|
133
+ def redis(db: :default)
134
+ redis_connections.with(db: db) do |connection|
104
135
  yield connection
105
136
  end
106
137
  end
@@ -108,8 +139,9 @@ module Stockpile
108
139
  # Accessor to connection pool. Defined on top level so it can be memoized
109
140
  # on the topmost level
110
141
  #
111
- # @return [ConnectionPool] ConnectionPool object from connection_pool gem
112
- def redis_connection_pool
113
- @redis_connection_pool ||= Stockpile::RedisConnection.connection_pool
142
+ # @return [Stockpile::RedisConnections] RedisConnections object holding all defined
143
+ # connection pools
144
+ def redis_connections
145
+ @redis_connections ||= Stockpile::RedisConnectionsFactory.build_connections
114
146
  end
115
147
  end
@@ -22,21 +22,32 @@ module Stockpile
22
22
  module Cache
23
23
  module_function
24
24
 
25
- def get(key:)
26
- value_from_cache = Stockpile.redis { |r| r.get(key) }
27
- Oj.load(value_from_cache) if value_from_cache
25
+ def get(db: :default, key:, compress: false)
26
+ value_from_cache = Stockpile.redis(db: db) { |r| r.get(key) }
27
+
28
+ return unless value_from_cache
29
+
30
+ if compress && value_from_cache
31
+ Oj.load(Zlib::Inflate.inflate(Base64.decode64(value_from_cache)))
32
+ else
33
+ Oj.load(value_from_cache)
34
+ end
28
35
  end
29
36
 
30
- def get_deferred(key:)
31
- sleep(Stockpile::SLUMBER_COOLDOWN) until Stockpile.redis { |r| r.exists(key) }
32
- value_from_cache = Stockpile.redis { |r| r.get(key) }
33
- Oj.load(value_from_cache)
37
+ def get_deferred(db: :default, key:, compress: false)
38
+ sleep(Stockpile::SLUMBER_COOLDOWN) until Stockpile.redis(db: db) { |r| r.exists(key) }
39
+
40
+ get(db: db, key: key, compress: compress)
34
41
  end
35
42
 
36
- def set(key:, payload:, ttl:)
37
- payload = Oj.dump(payload)
38
- Stockpile.redis { |r| r.set(key, payload) }
39
- Stockpile.redis { |r| r.expire(key, ttl) }
43
+ def set(db: :default, key:, payload:, ttl:, compress: false)
44
+ payload = if compress
45
+ Base64.encode64(Zlib::Deflate.deflate(Oj.dump(payload)))
46
+ else
47
+ Oj.dump(payload)
48
+ end
49
+
50
+ Stockpile.redis(db: db) { |r| r.setex(key, ttl, payload) }
40
51
  end
41
52
  end
42
53
  end
@@ -0,0 +1,28 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Copyright 2019 ConvertKit, LLC
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ module Stockpile
18
+ # == Stockpile::CachedValueExpirer
19
+ #
20
+ # Service class to wrap expiration of of cached value
21
+ module CachedValueExpirer
22
+ module_function
23
+
24
+ def expire_cached(db: :default, key:)
25
+ Stockpile.redis(db: db) { |r| r.expire(key, 0) }
26
+ end
27
+ end
28
+ end
@@ -22,11 +22,11 @@ module Stockpile
22
22
  module CachedValueReader
23
23
  module_function
24
24
 
25
- def read_or_yield(key:, ttl:, &block)
26
- if (result = Stockpile::Cache.get(key: key))
25
+ def read_or_yield(db: :default, key:, ttl:, &block)
26
+ if (result = Stockpile::Cache.get(db: db, key: key))
27
27
  result
28
28
  else
29
- Stockpile::Executor.perform(key: key, ttl: ttl, &block)
29
+ Stockpile::Executor.perform(db: db, key: key, ttl: ttl, &block)
30
30
  end
31
31
  end
32
32
  end
@@ -20,20 +20,25 @@ module Stockpile
20
20
  # Holds configuration for cache with writeable attributes allowing
21
21
  # dynamic change of configuration during runtime
22
22
  class Configuration
23
- attr_accessor :connection_pool, :connection_timeout, :lock_expiration,
24
- :redis_url, :sentinels, :slumber
23
+ attr_accessor :configuration_file, :connection_pool, :connection_timeout,
24
+ :lock_expiration, :redis_url, :sentinels, :slumber
25
25
 
26
26
  def initialize
27
+ @configuration_file = extract_configuration_file
27
28
  @connection_pool = extract_connection_pool
28
29
  @connection_timeout = extract_connection_timeout
29
30
  @lock_expiration = extract_lock_expiration
30
31
  @redis_url = extract_redis_url
31
- @sentinels = process_sentinels
32
+ @sentinels = extract_sentinels
32
33
  @slumber = extract_slumber
33
34
  end
34
35
 
35
36
  private
36
37
 
38
+ def extract_configuration_file
39
+ ENV.fetch('STOCKPILE_CONFIGURATION_FILE', nil)
40
+ end
41
+
37
42
  def extract_connection_pool
38
43
  ENV.fetch(
39
44
  'STOCKPILE_CONNECTION_POOL',
@@ -69,11 +74,10 @@ module Stockpile
69
74
  ).to_i
70
75
  end
71
76
 
72
- def process_sentinels
73
- ENV.fetch('STOCKPILE_REDIS_SENTINELS', '').split(',').map do |sentinel|
74
- host, port = sentinel.split(':')
75
- { host: host, port: port.to_i }
76
- end
77
+ def extract_sentinels
78
+ Stockpile::RedisConnectionsFactory.process_sentinels(
79
+ sentinels: ENV.fetch('STOCKPILE_REDIS_SENTINELS', '')
80
+ )
77
81
  end
78
82
  end
79
83
  end
@@ -23,5 +23,5 @@ module Stockpile
23
23
  DEFAULT_TTL = 60 * 5
24
24
  LOCK_PREFIX = 'stockpile_lock::'
25
25
  SLUMBER_COOLDOWN = 0.05
26
- VERSION = '1.0.0'
26
+ VERSION = '1.3.1'
27
27
  end
@@ -15,27 +15,36 @@
15
15
  # limitations under the License.
16
16
 
17
17
  module Stockpile
18
- # == Stockpile::RedisConnection
18
+ # == Stockpile::DefaultRedisConfiguration
19
19
  #
20
- # Wrapper around ConnectionPool and Redis to provide connectivity
21
- # to Redis with desired configuration and sane connection pool
22
- module RedisConnection
20
+ # Confiuration object for a single Redis database cache setup.
21
+ # Reads values out of environment, default values or uses
22
+ # configuration provided during runtime.
23
+ module DefaultRedisConfiguration
23
24
  module_function
24
25
 
25
- def connection_pool
26
- @connection_pool = ConnectionPool.new(connection_pool_options) do
27
- Redis.new(connection_options)
28
- end
26
+ def configuration
27
+ [
28
+ {
29
+ db: :default,
30
+ pool_configuration: pool_configuration,
31
+ redis_configuration: redis_configuration
32
+ }
33
+ ]
29
34
  end
30
35
 
31
- def connection_options
32
- { url: redis_url,
33
- sentinels: sentinels }
36
+ def redis_configuration
37
+ {
38
+ url: redis_url,
39
+ sentinels: sentinels
40
+ }.delete_if { |_k, v| v.nil? || v.empty? }
34
41
  end
35
42
 
36
- def connection_pool_options
37
- { size: pool_size,
38
- timeout: connection_timeout }
43
+ def pool_configuration
44
+ {
45
+ size: pool_size,
46
+ timeout: connection_timeout
47
+ }
39
48
  end
40
49
 
41
50
  def connection_timeout
@@ -22,13 +22,14 @@ module Stockpile
22
22
  # value to appear in cache instead. Will timeout after given amount of time
23
23
  # and will execute block if no value can be read from cache.
24
24
  class Executor
25
- attr_reader :key, :ttl
25
+ attr_reader :db, :key, :ttl
26
26
 
27
- def self.perform(key:, ttl:, &block)
28
- new(key, ttl).perform(&block)
27
+ def self.perform(db: :default, key:, ttl:, &block)
28
+ new(db, key, ttl).perform(&block)
29
29
  end
30
30
 
31
- def initialize(key, ttl)
31
+ def initialize(db, key, ttl)
32
+ @db = db
32
33
  @key = key
33
34
  @ttl = ttl
34
35
  end
@@ -43,17 +44,23 @@ module Stockpile
43
44
 
44
45
  private
45
46
 
47
+ def compress?
48
+ RedisConnections.compression?(db: db)
49
+ end
50
+
46
51
  def execution
47
- @execution ||= Stockpile::Lock.perform_locked(lock_key: lock_key) do
52
+ @execution ||= Stockpile::Lock.perform_locked(db: db, lock_key: lock_key) do
48
53
  yield
49
54
  end
50
55
  end
51
56
 
52
57
  def cache_and_release_execution
53
58
  Stockpile::Cache.set(
59
+ db: db,
54
60
  key: key,
55
61
  payload: execution.result,
56
- ttl: ttl
62
+ ttl: ttl,
63
+ compress: compress?
57
64
  )
58
65
 
59
66
  execution.release_lock
@@ -66,7 +73,7 @@ module Stockpile
66
73
 
67
74
  def wait_for_cache_or_yield
68
75
  Timeout.timeout(Stockpile.configuration.slumber) do
69
- Stockpile::Cache.get_deferred(key: key)
76
+ Stockpile::Cache.get_deferred(db: db, key: key, compress: compress?)
70
77
  end
71
78
  rescue Timeout::Error
72
79
  yield
@@ -23,13 +23,14 @@ module Stockpile
23
23
  # Stockpile::LockedExcutionResult will hold Stockpile::FailedLockExecution
24
24
  # as a result of execution
25
25
  class Lock
26
- attr_reader :lock_key
26
+ attr_reader :db, :lock_key
27
27
 
28
- def self.perform_locked(lock_key:, &block)
29
- new(lock_key).perform_locked(&block)
28
+ def self.perform_locked(db: :default, lock_key:, &block)
29
+ new(db, lock_key).perform_locked(&block)
30
30
  end
31
31
 
32
- def initialize(lock_key)
32
+ def initialize(db, lock_key)
33
+ @db = db
33
34
  @lock_key = lock_key
34
35
  end
35
36
 
@@ -44,7 +45,7 @@ module Stockpile
44
45
  private
45
46
 
46
47
  def failed_execution
47
- Stockpile::LockedExcutionResult.new(result: failed_lock, lock_key: lock_key)
48
+ Stockpile::LockedExcutionResult.new(db: db, result: failed_lock, lock_key: lock_key)
48
49
  end
49
50
 
50
51
  def failed_lock
@@ -52,11 +53,11 @@ module Stockpile
52
53
  end
53
54
 
54
55
  def lock
55
- Stockpile.redis { |r| r.set(lock_key, 1, nx: true, ex: Stockpile.configuration.lock_expiration) }
56
+ Stockpile.redis(db: db) { |r| r.set(lock_key, 1, nx: true, ex: Stockpile.configuration.lock_expiration) }
56
57
  end
57
58
 
58
59
  def successful_execution
59
- Stockpile::LockedExcutionResult.new(result: yield, lock_key: lock_key)
60
+ Stockpile::LockedExcutionResult.new(db: db, result: yield, lock_key: lock_key)
60
61
  end
61
62
  end
62
63
  end
@@ -19,15 +19,16 @@ module Stockpile
19
19
  #
20
20
  # Wrapper containing result of locked execution
21
21
  class LockedExcutionResult
22
- attr_reader :lock_key, :result
22
+ attr_reader :db, :lock_key, :result
23
23
 
24
- def initialize(lock_key:, result:)
24
+ def initialize(db: :default, lock_key:, result:)
25
+ @db = db
25
26
  @lock_key = lock_key
26
27
  @result = result
27
28
  end
28
29
 
29
30
  def release_lock
30
- Stockpile.redis { |r| r.expire(lock_key, 0) }
31
+ Stockpile.redis(db: db) { |r| r.expire(lock_key, 0) }
31
32
  end
32
33
 
33
34
  def success?
@@ -0,0 +1,35 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Copyright 2019 ConvertKit, LLC
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ module Stockpile
18
+ # == Stockpile::RedisConnections
19
+ #
20
+ # Wrapper around pools of Redis connections to allow multiple
21
+ # Redis database support
22
+ module RedisConnections
23
+ module_function
24
+
25
+ def compression?(db:)
26
+ instance_variable_get("@#{db}_compression".to_sym)
27
+ end
28
+
29
+ def with(db:)
30
+ instance_variable_get("@#{db}".to_sym).with do |connection|
31
+ yield connection
32
+ end
33
+ end
34
+ end
35
+ end
@@ -0,0 +1,55 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Copyright 2019 ConvertKit, LLC
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ module Stockpile
18
+ # == Stockpile::RedisConnectionsFactory
19
+ #
20
+ # Builds out connection pools out of provided configuration. Configurations
21
+ # are built with `*RedisConfiguration` classes. Providing a `.yml` file will
22
+ # override everything else and use that to build a config.
23
+ module RedisConnectionsFactory
24
+ module_function
25
+
26
+ def build_connections
27
+ configuration.each do |database|
28
+ pool = ConnectionPool.new(database[:pool_configuration]) do
29
+ Redis.new(database[:redis_configuration])
30
+ end
31
+
32
+ RedisConnections.instance_variable_set("@#{database[:db]}".to_sym, pool)
33
+
34
+ RedisConnections.instance_variable_set("@#{database[:db]}_compression".to_sym, database[:compression])
35
+ end
36
+
37
+ RedisConnections
38
+ end
39
+
40
+ def configuration
41
+ if Stockpile.configuration.configuration_file
42
+ Stockpile::YamlRedisConfiguration.configuration
43
+ else
44
+ Stockpile::DefaultRedisConfiguration.configuration
45
+ end
46
+ end
47
+
48
+ def process_sentinels(sentinels:)
49
+ sentinels.split(',').map do |sentinel|
50
+ host, port = sentinel.split(':')
51
+ { host: host, port: port.to_i }
52
+ end
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,72 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Copyright 2019 ConvertKit, LLC
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+
17
+ module Stockpile
18
+ # == Stockpile::YamlRedisConfiguration
19
+ #
20
+ # Confiuration object a multiple Redis database cache setup. Reads
21
+ # configuration out of provided `.yml` file.
22
+ module YamlRedisConfiguration
23
+ module_function
24
+
25
+ def configuration
26
+ parsed_configuration.map do |database, settings|
27
+ {
28
+ db: database,
29
+ pool_configuration: extract_pool(settings: settings),
30
+ redis_configuration: extract_redis(settings: settings),
31
+ compression: extract_compression(settings: settings)
32
+ }
33
+ end
34
+ end
35
+
36
+ def extract_compression(settings:)
37
+ return true if settings['compression'].eql?(true)
38
+
39
+ false
40
+ end
41
+
42
+ def extract_redis(settings:)
43
+ sentinels = Stockpile::RedisConnectionsFactory.process_sentinels(
44
+ sentinels: settings['sentinels'] || ''
45
+ )
46
+
47
+ {
48
+ url: settings['url'],
49
+ sentinels: sentinels
50
+ }.delete_if { |_k, v| v.nil? || v.empty? }
51
+ end
52
+
53
+ def extract_pool(settings:)
54
+ {
55
+ size: settings.dig('pool_options', 'size'),
56
+ timeout: settings.dig('pool_options', 'timeout')
57
+ }
58
+ end
59
+
60
+ def parsed_configuration
61
+ YAML.safe_load(
62
+ ERB.new(
63
+ raw_configuration
64
+ ).result
65
+ )
66
+ end
67
+
68
+ def raw_configuration
69
+ File.open(Stockpile.configuration.configuration_file).read
70
+ end
71
+ end
72
+ end
@@ -11,11 +11,12 @@ Gem::Specification.new do |spec|
11
11
  spec.authors = ['ConvertKit, LLC']
12
12
  spec.email = ['engineering@convertkit.com']
13
13
 
14
- spec.summary = 'Simple Redis based cache'
15
- spec.description = 'Cache with cache-stampede protection'
16
- spec.homepage = 'https://convertkit.com'
14
+ spec.summary = 'Redis based cache'
15
+ spec.description = 'Simple redis based cache with stampede protection'
17
16
  spec.license = 'Apache License Version 2.0'
18
17
 
18
+ spec.metadata['source_code_uri'] = 'https://github.com/ConvertKit/stockpile_cache'
19
+
19
20
  spec.files = `git ls-files | grep -Ev '^(spec)'`.split("\n")
20
21
 
21
22
  spec.executables = ['console']
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: stockpile_cache
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.3.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - ConvertKit, LLC
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2019-09-12 00:00:00.000000000 Z
11
+ date: 2020-06-01 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: connection_pool
@@ -66,7 +66,7 @@ dependencies:
66
66
  - - ">="
67
67
  - !ruby/object:Gem::Version
68
68
  version: '0'
69
- description: Cache with cache-stampede protection
69
+ description: Simple redis based cache with stampede protection
70
70
  email:
71
71
  - engineering@convertkit.com
72
72
  executables:
@@ -78,6 +78,7 @@ files:
78
78
  - ".gitignore"
79
79
  - ".rspec"
80
80
  - ".rubocop.yml"
81
+ - CHANGELOG.md
81
82
  - CODE_OF_CONDUCT.md
82
83
  - Gemfile
83
84
  - Gemfile.lock
@@ -88,20 +89,25 @@ files:
88
89
  - bin/setup
89
90
  - lib/stockpile.rb
90
91
  - lib/stockpile/cache.rb
92
+ - lib/stockpile/cached_value_expirer.rb
91
93
  - lib/stockpile/cached_value_reader.rb
92
94
  - lib/stockpile/configuration.rb
93
95
  - lib/stockpile/constants.rb
96
+ - lib/stockpile/default_redis_configuration.rb
94
97
  - lib/stockpile/executor.rb
95
98
  - lib/stockpile/failed_lock_execution.rb
96
99
  - lib/stockpile/lock.rb
97
100
  - lib/stockpile/locked_execution_result.rb
98
- - lib/stockpile/redis_connection.rb
101
+ - lib/stockpile/redis_connections.rb
102
+ - lib/stockpile/redis_connections_factory.rb
103
+ - lib/stockpile/yaml_redis_configuration.rb
99
104
  - lib/stockpile_cache.rb
100
105
  - stockpile-cache.gemspec
101
- homepage: https://convertkit.com
106
+ homepage:
102
107
  licenses:
103
108
  - Apache License Version 2.0
104
- metadata: {}
109
+ metadata:
110
+ source_code_uri: https://github.com/ConvertKit/stockpile_cache
105
111
  post_install_message:
106
112
  rdoc_options: []
107
113
  require_paths:
@@ -120,5 +126,5 @@ requirements: []
120
126
  rubygems_version: 3.0.3
121
127
  signing_key:
122
128
  specification_version: 4
123
- summary: Simple Redis based cache
129
+ summary: Redis based cache
124
130
  test_files: []