me-redis 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +10 -0
- data/.travis.yml +4 -0
- data/Gemfile +4 -0
- data/LICENSE.txt +21 -0
- data/README.md +408 -0
- data/Rakefile +10 -0
- data/bin/console +14 -0
- data/bin/setup +7 -0
- data/lib/me_redis.rb +189 -0
- data/lib/me_redis/hash.rb +29 -0
- data/lib/me_redis/integer.rb +7 -0
- data/lib/me_redis/me_redis_hot_migrator.rb +138 -0
- data/lib/me_redis/version.rb +3 -0
- data/lib/me_redis/zip_keys.rb +53 -0
- data/lib/me_redis/zip_to_hash.rb +92 -0
- data/lib/me_redis/zip_values.rb +146 -0
- data/me-redis.gemspec +36 -0
- metadata +137 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: a72bf4245ee1c3efe20256d83c4999e0002d78ea
|
4
|
+
data.tar.gz: 02acdfc1108cc6a3e06b04885b901fa1b63e7c95
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: cc66f942a6f49e9c86678ee7f92fdd2dbac60edd77df7b2a7feecde527769a0629da6f06b2d52bd7f5e9c1d9d9ac96a5905bc3ee17658e75d90336d31b5978e4
|
7
|
+
data.tar.gz: 5e699108252163cda89cfd604189343ad38c91a4dc4908d0aaefc890690415109ac6aa8d144a4846b350d20ffaf4e79c09292405fc69c3dffa30731ef6db7db9
|
data/.gitignore
ADDED
data/.travis.yml
ADDED
data/Gemfile
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2018 alekseyl
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,408 @@
|
|
1
|
+
# MeRedis
|
2
|
+
|
3
|
+
Me - Memory Efficient
|
4
|
+
|
5
|
+
This gem is delivering memory optimizations for Redis with slightest code changes.
|
6
|
+
|
7
|
+
To understand optimizations and how to use them
|
8
|
+
I suggest you to read my paper this topic: https://medium.com/p/61076c7da4c
|
9
|
+
|
10
|
+
#Features:
|
11
|
+
|
12
|
+
* seamless integration with code already in use, hardest integration possible:
|
13
|
+
add me_ prefix to some of your methods ( me_ methods implement hash memory optimization ).
|
14
|
+
It's all in MeRedis configuration, not your current code.
|
15
|
+
|
16
|
+
* hash key/value optimization with seamless code changes,
|
17
|
+
you can replace set('object:id', value) with me_set( 'object:id', value)
|
18
|
+
and free 90 byte for each ['object:id', value] pair.
|
19
|
+
|
20
|
+
* zips user-friendly key crumbs according to configuration, i.e. converts for example user:id to u:id
|
21
|
+
|
22
|
+
* zip integer parts of a keys with base62 encoding. Since all keys in redis are always strings, than we don't care for integers parts base, and by using base62 encoding we can 1.8 times shorten integer crumbs of keys
|
23
|
+
|
24
|
+
* respects pipelined and multi, properly works with Futures.
|
25
|
+
|
26
|
+
* allow different compressors for a different key namespaces,
|
27
|
+
you can deflate separately objects, short strings, large strings, primitives.
|
28
|
+
|
29
|
+
* hot migration module with fallbacks to previous keys.
|
30
|
+
|
31
|
+
* rails-less, it's rails independent, you can use it apart from rails
|
32
|
+
|
33
|
+
* seamless refactoring of old crumbs, i.e. you may rename crumbs keeping
|
34
|
+
existing cache intact
|
35
|
+
|
36
|
+
## Installation
|
37
|
+
|
38
|
+
Add this line to your application's Gemfile:
|
39
|
+
|
40
|
+
```ruby
|
41
|
+
gem 'me-redis'
|
42
|
+
```
|
43
|
+
|
44
|
+
And then execute:
|
45
|
+
|
46
|
+
$ bundle
|
47
|
+
|
48
|
+
Or install it yourself as:
|
49
|
+
|
50
|
+
$ gem install me-redis
|
51
|
+
|
52
|
+
## Usage
|
53
|
+
|
54
|
+
**Main consideration:**
|
55
|
+
1) less memory on redis side is better than less performance on ruby side
|
56
|
+
2) more result with less code changes,
|
57
|
+
i.e. overriding native redis methods with proper configuration basis is
|
58
|
+
preferred over mixin new methods
|
59
|
+
|
60
|
+
MeRedis based on three general optimization ideas:
|
61
|
+
* shorten keys
|
62
|
+
* compress values
|
63
|
+
* 'zip to hash', a Redis specific optimization from [Redis official memory optimization guide](https://medium.com/r/?url=https%3A%2F%2Fredis.io%2Ftopics%2Fmemory-optimization)
|
64
|
+
|
65
|
+
|
66
|
+
Thats why MeRedis contains three modules : MeRedis::ZipKeys, MeRedis::ZipValues, MeRedis::ZipToHash.
|
67
|
+
|
68
|
+
They can be used separately in any combination, but the simplest
|
69
|
+
way to deal with them all is call include upon MeRedis:
|
70
|
+
|
71
|
+
```ruby
|
72
|
+
Redis.include(MeRedis)
|
73
|
+
redis = Redis.new
|
74
|
+
```
|
75
|
+
|
76
|
+
If you want to keep a clear Redis class, you can do this way:
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
me_redis = Class.new(Redis).include(MeRedis).configure({...}).new
|
80
|
+
```
|
81
|
+
|
82
|
+
So now me_redis is a instance of unnamed class derived from Redis,
|
83
|
+
and patched with all MeRedis modules.
|
84
|
+
|
85
|
+
If you want to include them separately look at MeRedis.included method:
|
86
|
+
|
87
|
+
```ruby
|
88
|
+
def self.included(base)
|
89
|
+
base.prepend( MeRedis::ZipValues )
|
90
|
+
base.prepend( MeRedis::ZipKeys )
|
91
|
+
base.include( MeRedis::ZipToHash )
|
92
|
+
end
|
93
|
+
```
|
94
|
+
|
95
|
+
This is the right chain of prepending/including, so just remove unnecessary module.
|
96
|
+
|
97
|
+
###Base use
|
98
|
+
|
99
|
+
```ruby
|
100
|
+
redis = Redis.new
|
101
|
+
me_redis = Class.new(Redis).include(MeRedis).configure({
|
102
|
+
hash_max_ziplist_entries: 64,
|
103
|
+
zip_crumbs: :user,
|
104
|
+
integers_to_base62: true,
|
105
|
+
compress_namespaces: :user
|
106
|
+
}).new
|
107
|
+
|
108
|
+
# keep using code as you already do, like this:
|
109
|
+
me_redis.set( 'user:100', @user.to_json )
|
110
|
+
me_redis.get( 'user:100' )
|
111
|
+
|
112
|
+
# is equal under a hood to:
|
113
|
+
redis.set( 'u:1C', Zlib.deflate( @user.to_json ) )
|
114
|
+
Zlib.inflate( redis.get( 'u:1C' ) )
|
115
|
+
|
116
|
+
#OR replace all get/set/incr e.t.c with me_ prefixed version like this:
|
117
|
+
me_redis.me_set( 'user:100', @user.to_json )
|
118
|
+
me_redis.me_get( 'user:100' )
|
119
|
+
|
120
|
+
# under the hood equals to:
|
121
|
+
redis.hset( 'u:1', 'A', Zlib.deflate( @user.to_json ) )
|
122
|
+
Zlib.inflate( redis.hget( 'u:1', 'A' ) )
|
123
|
+
|
124
|
+
# future works same
|
125
|
+
ftr, me_ftr = nil, nil
|
126
|
+
me_redis.pipelined{ me_redis.set('user:100', '111'); me_ftr = me_redis.get(:user) }
|
127
|
+
#is equal to:
|
128
|
+
redis.pipelined{ redis.set( 'u:1C', Zlib.defalte( '111' ) ); ftr = redis.get('u:1C') }
|
129
|
+
# and
|
130
|
+
me_ftr.value == ftr.value
|
131
|
+
|
132
|
+
```
|
133
|
+
|
134
|
+
As you can see you can get a result with smallest or even none code changes!
|
135
|
+
|
136
|
+
All the ideas is to move complexity to config.
|
137
|
+
|
138
|
+
###Config
|
139
|
+
```ruby
|
140
|
+
|
141
|
+
Redis.include(MeRedis).configure( hash_max_ziplist_entries: 512 )
|
142
|
+
|
143
|
+
#Options are:
|
144
|
+
|
145
|
+
# if set - configures Redis hash_max_ziplist_entries value,
|
146
|
+
# otherwise it will be filled from Redis hash-max-ziplist-value
|
147
|
+
:hash_max_ziplist_entries
|
148
|
+
|
149
|
+
# if set - configures Redis hash_max_ziplist_entries value,
|
150
|
+
# otherwise it will be filled from Redis hash-max-ziplist-value
|
151
|
+
:hash_max_ziplist_value
|
152
|
+
|
153
|
+
# array or hash or string/sym of keys crumbs to zip,
|
154
|
+
# if a hash given it used as is,
|
155
|
+
# otherwise MeRedis tries to construct hash by using first char from each key given
|
156
|
+
# + integer in base62 starting from 1 for subsequent appearence of a crumbs starting with same chars
|
157
|
+
:zip_crumbs
|
158
|
+
|
159
|
+
# set to true if you want to zip ALL integers in keys to base62 form
|
160
|
+
:integers_to_base62
|
161
|
+
|
162
|
+
# regexp composed from zip_crumbs keys and general integer regexp (\d+) if integers_to_base62 is set
|
163
|
+
# better not to set directly
|
164
|
+
:key_zip_regxp
|
165
|
+
|
166
|
+
# keys prefixes/namespaces for values need to be zipped,
|
167
|
+
# acceptable formats:
|
168
|
+
# 1. single string/symbol/regexp - will map it to default compressor
|
169
|
+
# 2. array of string/symbols/regexp will map them all to default compressor
|
170
|
+
# 3. hash maps different kinds of 1 and 2 to custom compressors
|
171
|
+
|
172
|
+
# compress_namespaces will convert to two regexp:
|
173
|
+
# 1. one for strings and symbols
|
174
|
+
# 2. second for regexps
|
175
|
+
# they both will start with \A meaning this is a namespace/prefix
|
176
|
+
# be aware of it and omit \A in your regexps
|
177
|
+
:compress_namespaces
|
178
|
+
|
179
|
+
# if set directly than default_compressor applied to any key matched this regexp
|
180
|
+
# compress_namespaces is ignored
|
181
|
+
:compress_ns_regexp
|
182
|
+
|
183
|
+
# any kind of object which responds to compress/decompress methods
|
184
|
+
:default_compressor
|
185
|
+
```
|
186
|
+
###Config examples
|
187
|
+
|
188
|
+
```ruby
|
189
|
+
|
190
|
+
redis = Redis.include( MeRedis ).new
|
191
|
+
|
192
|
+
# zip key crumbs 'user', 'card', 'card_preview', to u, c, c1
|
193
|
+
# zips integer crumbs to base62,
|
194
|
+
# for keys starting with gz prefix compress values with Zlib
|
195
|
+
# for keys starting with json values with ActiveRecordJSONCompressor
|
196
|
+
Redis.configure(
|
197
|
+
hash_max_ziplist_entries: 256,
|
198
|
+
zip_crumbs: %i[user card card_preview json], # -> { user: :u, card: :c, card_preview: :c1 }
|
199
|
+
integers_to_base62: true,
|
200
|
+
compress_namespaces: {
|
201
|
+
gz: MeRedis::ZipValues::ZlibCompressor,
|
202
|
+
json: ActiveRecordJSONCompressor
|
203
|
+
}
|
204
|
+
)
|
205
|
+
|
206
|
+
redis.set( 'gz/card_preview:62', @card_preview )
|
207
|
+
|
208
|
+
#is equal under hood to:
|
209
|
+
redis.set( 'gz/c0:Z', Zlib.deflate( @card_preview) )
|
210
|
+
|
211
|
+
# and using me_ method:
|
212
|
+
redis.me_set( 'gz/card_preview:62', @card_preview )
|
213
|
+
|
214
|
+
#under the hood converts to:
|
215
|
+
redis.hset( 'gz/c0:1', '0', Zlib.deflate( @card_preview ) )
|
216
|
+
|
217
|
+
|
218
|
+
# It's possible to intersect zip_crumbs with compress_namespaces
|
219
|
+
Redis.configure(
|
220
|
+
hash_max_ziplist_entries: 256,
|
221
|
+
zip_crumbs: %i[user card card_preview json], # -> { user: :u, card: :c, card_preview: :c1 }
|
222
|
+
integers_to_base62: true,
|
223
|
+
compress_namespaces: {
|
224
|
+
gz: MeRedis::ZipValues::ZlibCompressor,
|
225
|
+
[:user, :card] => ActiveRecordJSONCompressor
|
226
|
+
}
|
227
|
+
)
|
228
|
+
|
229
|
+
redis.set( 'user:62', @user )
|
230
|
+
#under hood now converted to
|
231
|
+
redis.set( 'u:Z', ActiveRecordJSONCompressor.compress( @user ) )
|
232
|
+
|
233
|
+
#It's possible for compress_namespaces to use regexp:
|
234
|
+
Redis.configure(
|
235
|
+
zip_crumbs: %i[user card card_preview json], # -> { user: :u, card: :c, card_preview: :c1 }
|
236
|
+
compress_namespaces: {
|
237
|
+
/organization:[\d]+:card_preview/ => MeRedis::ZipValues::ZlibCompressor,
|
238
|
+
[:user, :card].map{|crumb| /organization:[\d]+:#{crumb}/ } => ActiveRecordJSONCompressor
|
239
|
+
}
|
240
|
+
)
|
241
|
+
|
242
|
+
redis.set( 'organization:1:user:62', @user )
|
243
|
+
#under hood now converted to
|
244
|
+
redis.set( 'organization:1:u:Z', ActiveRecordJSONCompressor.compress( @user ) )
|
245
|
+
|
246
|
+
# If you want intersect key zipping with regexp
|
247
|
+
# **you must intersect them using substituted crumbs!!!**
|
248
|
+
|
249
|
+
Redis.configure(
|
250
|
+
integers_to_base62: true,
|
251
|
+
zip_crumbs: %i[user card card_preview organization], # -> { user: :u, card: :c, card_preview: :c1, organization: :o }
|
252
|
+
compress_namespaces: {
|
253
|
+
/o:[a-zA-Z\d]+:card_preview/ => MeRedis::ZipValues::ZlibCompressor,
|
254
|
+
[:user, :card].map{|crumb| /o:[a-zA-Z\d]+:#{crumb}/ } => ActiveRecordJSONCompressor
|
255
|
+
}
|
256
|
+
)
|
257
|
+
|
258
|
+
redis.set( 'organization:1:user:62', @user )
|
259
|
+
#under hood now converted to
|
260
|
+
redis.set( 'o:1:u:Z', ActiveRecordJSONCompressor.compress( @user ) )
|
261
|
+
|
262
|
+
# You may set key zipping rules directly with a hash:
|
263
|
+
Redis.configure(
|
264
|
+
hash_max_ziplist_entries: 256,
|
265
|
+
zip_crumbs: { user: :u, card: :c, card_preview: :cp],
|
266
|
+
integers_to_base62: true,
|
267
|
+
)
|
268
|
+
|
269
|
+
# This config means: don't zip keys only zip values.
|
270
|
+
# For keys started with :user, :card, :card_preview
|
271
|
+
# compress all values with default compressor
|
272
|
+
# default compressor is ZlibCompressor if you prepend ZipValues module or include whole MeRedis module,
|
273
|
+
# otherwise it is EmptyCompressor which doesn't compress anything
|
274
|
+
Redis.configure(
|
275
|
+
hash_max_ziplist_entries: 256,
|
276
|
+
compress_namespaces: %i[user card card_preview]
|
277
|
+
)
|
278
|
+
```
|
279
|
+
Now I may suggest some best practices for MeRedis configure:
|
280
|
+
|
281
|
+
* explicit crumbs schema is preferable over implicit
|
282
|
+
* if you are going lazy, and use implicit schemas, than avoid keys shuffling,
|
283
|
+
cause it messes with your cache
|
284
|
+
* better to configure hash-max-ziplist-* in MeRedis.configure than elsewhere.
|
285
|
+
* use in persistent Redis-based system with extreme caution
|
286
|
+
|
287
|
+
|
288
|
+
#Custom Compressors
|
289
|
+
|
290
|
+
MeRedis allow you to compress values through different compressor.
|
291
|
+
Here is an example of custom compressor for ActiveRecord objects,
|
292
|
+
I use to test compression ratio against plain compression of to_json.
|
293
|
+
|
294
|
+
```ruby
|
295
|
+
|
296
|
+
module ActiveRecordJSONCompressor
|
297
|
+
# this is the example, automated for simplicity, if DB schema changes, than cache may broke!!
|
298
|
+
# in reallife scenario either invalidate cache, or use explicit schemas
|
299
|
+
# like User: { first_name: 1, last_name: 2 ... },
|
300
|
+
# than your cache will be safer on schema changes.
|
301
|
+
COMPRESSOR_SCHEMAS = [User, HTag].map{|mdl|
|
302
|
+
[mdl.to_s, mdl.column_names.each_with_index.map{ |el, i| [el, (20 + i).to_base62] }.to_h]
|
303
|
+
}.to_h.with_indifferent_access
|
304
|
+
|
305
|
+
REVERSE_COMPRESSOR_SCHEMA = COMPRESSOR_SCHEMAS.dup.transform_values(&:invert)
|
306
|
+
|
307
|
+
def self.compress( object )
|
308
|
+
use_schema = COMPRESSOR_SCHEMAS[object.class.to_s]
|
309
|
+
# _s - shorten for schema, s cannot be used since its a number in Base62 system
|
310
|
+
Zlib.deflate(
|
311
|
+
object.serializable_hash
|
312
|
+
.slice( *use_schema.keys )
|
313
|
+
.transform_keys{ |k| use_schema[k] }
|
314
|
+
.reject{ |_,v| v.blank? }
|
315
|
+
.merge!( _s: object.class.to_s ).to_json
|
316
|
+
)
|
317
|
+
end
|
318
|
+
|
319
|
+
def self.decompress(value)
|
320
|
+
compressed_hash = JSON.load( Zlib.inflate(value) )
|
321
|
+
model = compressed_hash.delete('_s')
|
322
|
+
schema = REVERSE_COMPRESSOR_SCHEMA[model]
|
323
|
+
model.constantize.new( compressed_hash.transform_keys{ |k| schema[k] } )
|
324
|
+
end
|
325
|
+
end
|
326
|
+
|
327
|
+
```
|
328
|
+
|
329
|
+
#Hot migration
|
330
|
+
MeRedis deliver additional module for hot migration to KeyZipping and ZipToHash.
|
331
|
+
We don't need one in generally for base implementation of ZipValues cause
|
332
|
+
its getter methods fallbacks to value.
|
333
|
+
|
334
|
+
###Features
|
335
|
+
* mget hget hgetall get exists type getset - fallbacks for key_zipping
|
336
|
+
* me_get me_mget - fallbacks for hash zipping
|
337
|
+
* partially respects pipelining and multi
|
338
|
+
* protecting you from accidentally do many to less many migration
|
339
|
+
and from ZipToHash migration without key zipping (
|
340
|
+
though it's impossible to hot migrate from 'user:100' to 'user:1', 'B',
|
341
|
+
because of same namespace 'user' for flat key/value pair and hashes,
|
342
|
+
you'll definetely get an error )
|
343
|
+
* reverse migration methods
|
344
|
+
|
345
|
+
```ruby
|
346
|
+
redis = Redis.include( MeRedisHotMigrator ).configure(
|
347
|
+
zip_crumbs: :user
|
348
|
+
)
|
349
|
+
|
350
|
+
usr_1_cache = redis.me_get('user:1')
|
351
|
+
|
352
|
+
all_user_keys = redis.keys('user*')
|
353
|
+
redis.migrate_to_hash_representation( all_user_keys )
|
354
|
+
|
355
|
+
usr_1_cache == redis.me_get('user:1') # true
|
356
|
+
|
357
|
+
redis.reverse_from_hash_representation!( all_user_keys )
|
358
|
+
|
359
|
+
usr_1_cache == redis.me_get('user:1') # true
|
360
|
+
|
361
|
+
```
|
362
|
+
|
363
|
+
For persistent store use with extreme caution!!
|
364
|
+
Backup, test, test, user test and after you are sure than you may migrate.
|
365
|
+
|
366
|
+
Try not to stuck with it because doing double amount of actions,
|
367
|
+
do BG deploy of code, run migration in parallel, replace MeRedisHotMigrator with MeRedis
|
368
|
+
do BG deploy and you are done.
|
369
|
+
|
370
|
+
#Limitations
|
371
|
+
|
372
|
+
###Me_* methods limitation
|
373
|
+
|
374
|
+
Some of me_methods like me_mget/me_mset/me_getset
|
375
|
+
are imitations for corresponded base methods behaviour through
|
376
|
+
pipeline and transactions. So inside pipelined call it may not
|
377
|
+
deliver a completely equal behaviour.
|
378
|
+
|
379
|
+
me_mget has an additional double me_mget_p in case you need to use it with futures.
|
380
|
+
|
381
|
+
###ZipKeys and ZipValues
|
382
|
+
As I already mention if you want to use custom prefix regex
|
383
|
+
for zipping values than it must be constructed with a crumbs substitutions,
|
384
|
+
not the original crumb, see config example.
|
385
|
+
|
386
|
+
```ruby
|
387
|
+
|
388
|
+
```
|
389
|
+
|
390
|
+
## Development
|
391
|
+
|
392
|
+
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
393
|
+
|
394
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
395
|
+
|
396
|
+
## Contributing
|
397
|
+
|
398
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/alekseyl/me-redis.
|
399
|
+
|
400
|
+
|
401
|
+
## License
|
402
|
+
|
403
|
+
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
404
|
+
|
405
|
+
## ToDo List
|
406
|
+
|
407
|
+
* add keys method
|
408
|
+
* refactor readme
|
data/Rakefile
ADDED
data/bin/console
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
require "bundler/setup"
|
4
|
+
require "me_redis"
|
5
|
+
|
6
|
+
# You can add fixtures and/or initialization code here to make experimenting
|
7
|
+
# with your gem easier. You can also use a different console, if you like.
|
8
|
+
|
9
|
+
# (If you use this, don't forget to add pry to your Gemfile!)
|
10
|
+
# require "pry"
|
11
|
+
# Pry.start
|
12
|
+
|
13
|
+
require "irb"
|
14
|
+
IRB.start
|
data/bin/setup
ADDED
data/lib/me_redis.rb
ADDED
@@ -0,0 +1,189 @@
|
|
1
|
+
require 'me_redis/version'
|
2
|
+
require 'me_redis/zip_keys'
|
3
|
+
require 'me_redis/zip_to_hash'
|
4
|
+
require 'me_redis/zip_values'
|
5
|
+
require 'me_redis/me_redis_hot_migrator'
|
6
|
+
require 'me_redis/integer'
|
7
|
+
require 'me_redis/hash'
|
8
|
+
|
9
|
+
require 'zlib'
|
10
|
+
|
11
|
+
# Main ideas:
|
12
|
+
# 1) less memory on redis is better than performance on ruby code
|
13
|
+
# 2) more result with less code changes,
|
14
|
+
# i.e. overriding old methods with proper configure is preferred over mixin new methods
|
15
|
+
# 3) rails-less
|
16
|
+
|
17
|
+
module MeRedis
|
18
|
+
module ClassMethods
|
19
|
+
|
20
|
+
def configure( config = nil )
|
21
|
+
# at start they are nils, but at subsequent calls they may not be nils
|
22
|
+
me_config.key_zip_regxp = nil
|
23
|
+
me_config.compress_ns_regexp = nil
|
24
|
+
@zip_ns_finder = nil
|
25
|
+
|
26
|
+
config.each{ |key,value| me_config.send( "#{key}=", value ) } if config
|
27
|
+
|
28
|
+
yield( me_config ) if block_given?
|
29
|
+
|
30
|
+
|
31
|
+
prepare_zip_crumbs
|
32
|
+
prepare_compressors
|
33
|
+
|
34
|
+
# useful for chaining with dynamic class creations
|
35
|
+
self
|
36
|
+
end
|
37
|
+
|
38
|
+
def me_config
|
39
|
+
@me_config ||= Struct.new(
|
40
|
+
# if set - configures Redis hash_max_ziplist_entries value,
|
41
|
+
# otherwise it will be filled from Redis hash-max-ziplist-value
|
42
|
+
:hash_max_ziplist_entries,
|
43
|
+
# same as above only for value, only resets it globally if present
|
44
|
+
:hash_max_ziplist_value,
|
45
|
+
# array or hash or string/sym of key crumbs to zip, if a hash given it used as is,
|
46
|
+
# otherwise meredis tries to construct hash by using first char from each key + integer in base62 form for
|
47
|
+
# subsequent appearence of a crumb starting with same char
|
48
|
+
:zip_crumbs,
|
49
|
+
# zip integers in keys to base62 form
|
50
|
+
:integers_to_base62,
|
51
|
+
# regex composed from zip_crumbs keys and integer regexp if integers_to_base62 is set
|
52
|
+
:key_zip_regxp,
|
53
|
+
# prefixes/namespaces for keys need zipping,
|
54
|
+
# acceptable formats:
|
55
|
+
# 1. single string/sym will map it to defauilt compressor
|
56
|
+
# 2. array of string/syms will map it to defauilt compressor
|
57
|
+
# 3. hash maps different kinds of 1 and 2 to custom compressors
|
58
|
+
:compress_namespaces,
|
59
|
+
# if configured than default_compressor used for compression of all keys matched and compress_namespaces is ignored
|
60
|
+
:compress_ns_regexp,
|
61
|
+
|
62
|
+
:default_compressor
|
63
|
+
).new(512)
|
64
|
+
end
|
65
|
+
|
66
|
+
def zip_crumbs; me_config.zip_crumbs end
|
67
|
+
|
68
|
+
def key_zip_regxp
|
69
|
+
return me_config.key_zip_regxp if me_config.key_zip_regxp
|
70
|
+
regexp_parts = []
|
71
|
+
#reverse order just to be sure we replaced longer strings before shorter
|
72
|
+
# also we need to sort by length, not just sort, because we must try to replace 'z_key_a' first,
|
73
|
+
# and only after that we can replace 'key'
|
74
|
+
regexp_parts << "(#{zip_crumbs.keys.sort_by(&:length).reverse.join('|')})" if zip_crumbs
|
75
|
+
regexp_parts << '(\d+)' if me_config.integers_to_base62
|
76
|
+
me_config.key_zip_regxp ||= /#{regexp_parts.join('|')}/
|
77
|
+
end
|
78
|
+
|
79
|
+
def get_compressor_namespace_from_key( key )
|
80
|
+
ns_matched = zip_ns_finder[:rgxps_ns] && key.match(zip_ns_finder[:rgxps_ns])
|
81
|
+
if ns_matched&.captures
|
82
|
+
zip_ns_finder[:rgxps_arr][ns_matched.captures.each_with_index.find{|el,i| el}[1]]
|
83
|
+
else
|
84
|
+
zip_ns_finder[:string_ns] && key.match(zip_ns_finder[:string_ns])&.send(:[], 0)
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
def zip?(key)
|
89
|
+
me_config.compress_ns_regexp&.match?(key) ||
|
90
|
+
zip_ns_finder[:string_ns]&.match?(key) ||
|
91
|
+
zip_ns_finder[:rgxps_ns]&.match?(key)
|
92
|
+
end
|
93
|
+
|
94
|
+
def zip_ns_finder
|
95
|
+
return @zip_ns_finder if @zip_ns_finder
|
96
|
+
regexps_compress_ns = me_config.compress_namespaces.keys.select{|key| key.is_a?(Regexp) }
|
97
|
+
strs_compress_ns = me_config.compress_namespaces.keys.select{|key| !key.is_a?(Regexp) }
|
98
|
+
|
99
|
+
@zip_ns_finder = {
|
100
|
+
string_ns: strs_compress_ns.length == 0 ? nil : /\A(#{strs_compress_ns.sort_by(&:length).reverse.join('|')})/,
|
101
|
+
rgxps_ns: regexps_compress_ns.length == 0 ? nil : /\A#{regexps_compress_ns.map{|rgxp| "(#{rgxp})" }.join('|')}/,
|
102
|
+
rgxps_arr: regexps_compress_ns
|
103
|
+
}
|
104
|
+
end
|
105
|
+
|
106
|
+
def get_compressor_for_key( key )
|
107
|
+
if me_config.compress_ns_regexp
|
108
|
+
me_config.default_compressor
|
109
|
+
else
|
110
|
+
me_config.compress_namespaces[get_compressor_namespace_from_key( key )]
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
private
|
115
|
+
|
116
|
+
def prepare_zip_crumbs
|
117
|
+
if zip_crumbs.is_a?( Array )
|
118
|
+
result = {}
|
119
|
+
me_config.zip_crumbs.map!(&:to_s).each do |sub|
|
120
|
+
if result[sub[0]]
|
121
|
+
i = 0
|
122
|
+
begin i += 1 end while( result["#{sub[0]}#{i.to_base62}"] )
|
123
|
+
result["#{sub[0]}#{i.to_base62}"] = sub.to_s
|
124
|
+
else
|
125
|
+
result[sub[0]] = sub
|
126
|
+
end
|
127
|
+
end
|
128
|
+
me_config.zip_crumbs = result.invert
|
129
|
+
elsif zip_crumbs.is_a?( String ) || zip_crumbs.is_a?( Symbol )
|
130
|
+
me_config.zip_crumbs = { me_config.zip_crumbs.to_s => me_config.zip_crumbs[0] }
|
131
|
+
elsif zip_crumbs.is_a?( Hash )
|
132
|
+
me_config.zip_crumbs = zip_crumbs.transform_keys(&:to_s).transform_values(&:to_s)
|
133
|
+
raise ArgumentError.new("pack subs cannot be inverted properly.
|
134
|
+
repack subs: #{zip_crumbs}, repack keys invert: #{zip_crumbs.invert}") unless zip_crumbs.invert.invert == zip_crumbs
|
135
|
+
elsif zip_crumbs
|
136
|
+
raise ArgumentError.new("Wrong class for zip_crumbs, expected Array, Hash, String or Symbol! Got: #{zip_crumbs.class.to_s}")
|
137
|
+
end
|
138
|
+
|
139
|
+
key_zip_regxp
|
140
|
+
end
|
141
|
+
|
142
|
+
def prepare_compressors
|
143
|
+
|
144
|
+
me_config.default_compressor ||= MeRedis::ZipValues::EmptyCompressor
|
145
|
+
|
146
|
+
me_config.compress_namespaces = case me_config.compress_namespaces
|
147
|
+
when Array
|
148
|
+
me_config.compress_namespaces.map{|ns| [replace_ns(ns), me_config.default_compressor] }.to_h
|
149
|
+
when String, Symbol, Regexp
|
150
|
+
{ replace_ns( me_config.compress_namespaces ) => me_config.default_compressor }
|
151
|
+
when Hash
|
152
|
+
me_config.compress_namespaces.inject({}) do |sum, (name_space, compressor)|
|
153
|
+
name_space.is_a?( Array ) ?
|
154
|
+
sum.merge!( name_space.map{ |ns| [replace_ns( ns), compressor] }.to_h )
|
155
|
+
: sum[replace_ns(name_space)] = compressor
|
156
|
+
sum
|
157
|
+
end
|
158
|
+
else
|
159
|
+
raise ArgumentError.new(<<~NS_ERR) if me_config.compress_namespaces
|
160
|
+
Wrong class for compress_namespaces, expected Array,
|
161
|
+
Hash, String or Symbol! Got: #{me_config.compress_namespaces.class.to_s}
|
162
|
+
NS_ERR
|
163
|
+
{}
|
164
|
+
end
|
165
|
+
|
166
|
+
zip_ns_finder
|
167
|
+
end
|
168
|
+
|
169
|
+
def replace_ns(ns)
|
170
|
+
( zip_crumbs && zip_crumbs[ns.to_s] ) || ( check_ns_type!(ns) && ( ns.is_a?(Regexp) ? ns : ns.to_s ) )
|
171
|
+
end
|
172
|
+
|
173
|
+
def check_ns_type!( ns )
|
174
|
+
case ns
|
175
|
+
when String, Symbol, Regexp
|
176
|
+
true
|
177
|
+
else
|
178
|
+
raise 'Must be Symbol, String or Regexp!'
|
179
|
+
end
|
180
|
+
end
|
181
|
+
end
|
182
|
+
|
183
|
+
#include
|
184
|
+
def self.included(base)
|
185
|
+
base.prepend( MeRedis::ZipValues )
|
186
|
+
base.prepend( MeRedis::ZipKeys )
|
187
|
+
base.include( MeRedis::ZipToHash )
|
188
|
+
end
|
189
|
+
end
|
@@ -0,0 +1,29 @@
|
|
1
|
+
class Hash
|
2
|
+
# Returns a new hash with all keys converted using the +block+ operation.
|
3
|
+
#
|
4
|
+
# hash = { name: 'Rob', age: '28' }
|
5
|
+
#
|
6
|
+
# hash.transform_keys { |key| key.to_s.upcase } # => {"NAME"=>"Rob", "AGE"=>"28"}
|
7
|
+
#
|
8
|
+
# If you do not provide a +block+, it will return an Enumerator
|
9
|
+
# for chaining with other methods:
|
10
|
+
#
|
11
|
+
# hash.transform_keys.with_index { |k, i| [k, i].join } # => {"name0"=>"Rob", "age1"=>"28"}
|
12
|
+
def transform_keys
|
13
|
+
return enum_for(:transform_keys) { size } unless block_given?
|
14
|
+
result = {}
|
15
|
+
each_key do |key|
|
16
|
+
result[yield(key)] = self[key]
|
17
|
+
end
|
18
|
+
result
|
19
|
+
end
|
20
|
+
# Destructively converts all keys using the +block+ operations.
|
21
|
+
# Same as +transform_keys+ but modifies +self+.
|
22
|
+
def transform_keys!
|
23
|
+
return enum_for(:transform_keys!) { size } unless block_given?
|
24
|
+
keys.each do |key|
|
25
|
+
self[yield(key)] = delete(key)
|
26
|
+
end
|
27
|
+
self
|
28
|
+
end
|
29
|
+
end unless Hash.method_defined?(:transform_keys)
|
@@ -0,0 +1,138 @@
|
|
1
|
+
#We need only to fallback getters, when you are setting
|
2
|
+
# new value it will go in a new place already
|
3
|
+
# me_mget doesn't compartible with pipeline, it will raise exception when placed inside one.
|
4
|
+
module MeRedisHotMigrator
|
5
|
+
ZK_FALLBACK_METHODS = %i[mget hget hgetall get exists type getset]
|
6
|
+
|
7
|
+
def self.included(base)
|
8
|
+
base::Future.prepend(FutureMigrator)
|
9
|
+
|
10
|
+
base.class_eval do
|
11
|
+
ZK_FALLBACK_METHODS.each do |method|
|
12
|
+
alias_method "_#{method}", method
|
13
|
+
end
|
14
|
+
|
15
|
+
include(MeRedis)
|
16
|
+
|
17
|
+
def me_get( key )
|
18
|
+
prev_future = _get( key ) unless @client.is_a?(self.class::Client)
|
19
|
+
newvl = super(key)
|
20
|
+
|
21
|
+
newvl.prev_future = prev_future if newvl.is_a?(self.class::Future)
|
22
|
+
newvl || _get( key )
|
23
|
+
end
|
24
|
+
|
25
|
+
def me_mget(*keys)
|
26
|
+
#cannot run in pipeline because of fallbacks
|
27
|
+
raise 'Cannot run in pipeline!!!' unless @client.is_a?(self.class::Client)
|
28
|
+
me_mget_p(*keys).map(&:value)
|
29
|
+
end
|
30
|
+
|
31
|
+
end
|
32
|
+
|
33
|
+
base.prepend( MeRedisHotMigrator::PrependMethods )
|
34
|
+
end
|
35
|
+
|
36
|
+
module PrependMethods
|
37
|
+
ZK_FALLBACK_METHODS.each do |method|
|
38
|
+
define_method(method) do |*args|
|
39
|
+
prev_future = send("_#{method}", *args) unless @client.is_a?(self.class::Client)
|
40
|
+
newvl = super(*args)
|
41
|
+
|
42
|
+
newvl.prev_future = prev_future if newvl.is_a?(self.class::Future)
|
43
|
+
|
44
|
+
if method != :mget
|
45
|
+
newvl || send("_#{method}", *args)
|
46
|
+
else
|
47
|
+
newvl.is_a?(Array) ? newvl.zip( send("_#{method}", *args) ).map!{|nvl, oldv| nvl || oldv } : newvl
|
48
|
+
end
|
49
|
+
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
|
55
|
+
#-------------------------------ME method migration------------------------------
|
56
|
+
#check if migration possible, if not raises exception with a reason
|
57
|
+
def hash_migration_possible?( keys )
|
58
|
+
result = keys.map{ |key| [split_key(key).each_with_index.map{|v,i| i == 0 ? zip_key(v) : v }, key] }.to_h
|
59
|
+
|
60
|
+
raise ArgumentError.new( "Hash zipping is not one to one! #{result.keys} != #{keys}" ) if result.length != keys.length
|
61
|
+
|
62
|
+
result.each do |sp_key, key|
|
63
|
+
key_start = key.to_s.scan(/\A(.*?)(\d+)\z/).flatten[0]
|
64
|
+
if sp_key[0].start_with?( key_start )
|
65
|
+
raise ArgumentError.new( "#{sp_key[0]} contains original key main part: #{key_start} Hash migration must be done with key zipping!")
|
66
|
+
end
|
67
|
+
end
|
68
|
+
|
69
|
+
true
|
70
|
+
end
|
71
|
+
|
72
|
+
# keys will exists after migrate, you need to call del(keys) directly
|
73
|
+
# uses hsetnx, meaning you will not overwrtite new values
|
74
|
+
def migrate_to_hash_representation( keys )
|
75
|
+
raise StandardError.new('Cannot migrate inside pipeline.') unless @client.is_a?( self.class::Client )
|
76
|
+
raise ArgumentError.new('Migration is unavailable!') unless hash_migration_possible?( keys )
|
77
|
+
|
78
|
+
values = mget( keys )
|
79
|
+
pipelined do
|
80
|
+
keys.each_with_index do |key, i|
|
81
|
+
me_setnx( key, values[i] )
|
82
|
+
end
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def reverse_from_hash_representation!( keys )
|
87
|
+
raise "Cannot migrate inside pipeline" unless @client.is_a?(self.class::Client )
|
88
|
+
values = me_mget( keys )
|
89
|
+
|
90
|
+
pipelined do
|
91
|
+
keys.each_with_index{|key, i| set( key, values[i] ) }
|
92
|
+
end
|
93
|
+
end
|
94
|
+
#-------------------------------ME method migration ENDED------------------------
|
95
|
+
|
96
|
+
# -------------------------------KZ migration------------------------------------
|
97
|
+
def migrate_to_key_zipping(keys)
|
98
|
+
pipelined do
|
99
|
+
zk_map_keys(keys).each{|new_key, key| renamenx( key, new_key )}
|
100
|
+
end
|
101
|
+
end
|
102
|
+
|
103
|
+
# reverse migration done with same set of keys, i.e,
|
104
|
+
# if you migrated [ user:1, user:2 ] with migrate_to_key_zipping and want to reverse migration
|
105
|
+
# then use same argument [ user:1, user:2 ]
|
106
|
+
def reverse_from_key_zipping!( keys )
|
107
|
+
pipelined do
|
108
|
+
zk_map_keys(keys).each{|new_key, key| rename( new_key, key ) }
|
109
|
+
end
|
110
|
+
end
|
111
|
+
|
112
|
+
# use only uniq keys! or zk_map_keys will fire an error!
|
113
|
+
# if transition is not one to one zk_map_keys would also fire an error
|
114
|
+
def zk_map_keys(keys)
|
115
|
+
keys.map{ |key| [zip_key(key), key] }.to_h
|
116
|
+
.tap{ |result| raise ArgumentError.new( "Key zipping is not one to one! #{result.keys} != #{keys}" ) if result.length != keys.length }
|
117
|
+
end
|
118
|
+
|
119
|
+
def key_zipping_migration_reversible?( keys )
|
120
|
+
!!zk_map_keys(keys)
|
121
|
+
end
|
122
|
+
# -------------------------------KZ migration ENDED ------------------------------------
|
123
|
+
|
124
|
+
module FutureMigrator
|
125
|
+
def prev_future=(new_prev_future); @prev_future = new_prev_future end
|
126
|
+
def value;
|
127
|
+
vl = super
|
128
|
+
if !vl
|
129
|
+
@prev_future&.value
|
130
|
+
elsif vl.is_a?(Array) && @prev_future
|
131
|
+
vl.zip( @prev_future&.value ).map{|nvl, old| nvl || old }
|
132
|
+
else
|
133
|
+
vl
|
134
|
+
end
|
135
|
+
end
|
136
|
+
end
|
137
|
+
|
138
|
+
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
module MeRedis
|
2
|
+
# how to use:
|
3
|
+
# Redis.prepend( MeRedis::KeyMinimizer )
|
4
|
+
module ZipKeys
|
5
|
+
|
6
|
+
def self.prepended(base)
|
7
|
+
base.extend( MeRedis::ClassMethods )
|
8
|
+
end
|
9
|
+
|
10
|
+
def zip_key( key )
|
11
|
+
key.to_s.split( self.class.key_zip_regxp ).map do |zip_me|
|
12
|
+
if zip_me.to_i != 0
|
13
|
+
zip_me.to_i.to_base62
|
14
|
+
else
|
15
|
+
self.class.zip_crumbs&.send(:[], zip_me ) || zip_me
|
16
|
+
end
|
17
|
+
end.join
|
18
|
+
end
|
19
|
+
|
20
|
+
#---- h_methods ---------------------------
|
21
|
+
def hdel( key, hkey ); super( zip_key(key), hkey ) end
|
22
|
+
def hset( key, hkey, value ); super( zip_key(key), hkey, value ) end
|
23
|
+
def hsetnx( key, hkey, value ); super( zip_key(key), hkey, value ) end
|
24
|
+
def hexists( key, hkey ); super( zip_key(key), hkey ) end
|
25
|
+
def hget( key, hkey ); super( zip_key(key), hkey ) end
|
26
|
+
def hincrby( key, hkey, value ); super( zip_key(key), hkey, value ) end
|
27
|
+
def hmset( key, *args ); super( zip_key(key), *args ) end
|
28
|
+
def hmget( key, *args ); super( zip_key(key), *args ) end
|
29
|
+
#---- Hash methods END --------------------
|
30
|
+
|
31
|
+
|
32
|
+
def incr( key ); super( zip_key(key) ) end
|
33
|
+
def get( key ); super( zip_key(key) ) end
|
34
|
+
|
35
|
+
def exists(key); super( zip_key(key) ) end
|
36
|
+
def type(key); super(zip_key(key)) end
|
37
|
+
def decr(key); super(zip_key(key)) end
|
38
|
+
def persist(key); super(zip_key(key)) end
|
39
|
+
|
40
|
+
def decrby( key, decrement ); super(zip_key(key), decrement) end
|
41
|
+
def set( key, value ); super( zip_key(key), value ) end
|
42
|
+
def mset( *key_values ); super( *key_values.each_slice(2).map{ |k,v| [zip_key(k),v] }.flatten ) end
|
43
|
+
def mget( *keys ); super( *keys.map!{ |k| zip_key(k) } ) end
|
44
|
+
|
45
|
+
def getset( key, value ); super( zip_key(key), value ) end
|
46
|
+
def move(key, db); super( zip_key(key), db ) end
|
47
|
+
|
48
|
+
def del(*keys); super( *keys.map{ |key| zip_key(key) } ) end
|
49
|
+
|
50
|
+
def rename(old_name, new_name); super( zip_key(old_name), zip_key(new_name) ) end
|
51
|
+
def renamenx(old_name, new_name); super( zip_key(old_name), zip_key(new_name) ) end
|
52
|
+
end
|
53
|
+
end
|
@@ -0,0 +1,92 @@
|
|
1
|
+
module MeRedis
|
2
|
+
# include
|
3
|
+
module ZipToHash
|
4
|
+
|
5
|
+
module PrependMethods
|
6
|
+
def initialize(*args, &block)
|
7
|
+
super(*args, &block)
|
8
|
+
|
9
|
+
# hash-max-ziplist-entries must be cashed, we can't ask Redis every time we need to zip keys,
|
10
|
+
# cause it's less performant and impossible during pipelining.
|
11
|
+
_config = config(:get, 'hash-max-ziplist-*' )
|
12
|
+
@hash_max_ziplist_entries = _config['hash-max-ziplist-entries'].to_i
|
13
|
+
if self.class.me_config.hash_max_ziplist_entries && @hash_max_ziplist_entries != self.class.me_config.hash_max_ziplist_entries
|
14
|
+
#if me_config configures hash-max-ziplist-entries than we assume it global
|
15
|
+
config(:set, 'hash-max-ziplist-entries', self.class.me_config.hash_max_ziplist_entries )
|
16
|
+
end
|
17
|
+
|
18
|
+
if self.class.me_config.hash_max_ziplist_value &&
|
19
|
+
self.class.me_config.hash_max_ziplist_value != _config['hash-max-ziplist-value'].to_i
|
20
|
+
|
21
|
+
config(:set, 'hash-max-ziplist-value', self.class.me_config.hash_max_ziplist_value)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
def config(action, *args)
|
26
|
+
@hash_max_ziplist_entries = args[1].to_i if action.to_s == 'set' && args[0] == 'hash-max-ziplist-entries'
|
27
|
+
super( action, *args )
|
28
|
+
end
|
29
|
+
|
30
|
+
end
|
31
|
+
|
32
|
+
def self.included(base)
|
33
|
+
base.extend( MeRedis::ClassMethods )
|
34
|
+
base.prepend( PrependMethods )
|
35
|
+
end
|
36
|
+
|
37
|
+
def me_del( *keys )
|
38
|
+
keys.length == 1 ? hdel( *split_key(*keys) ) : pipelined{ keys.each{ |key| hdel( *split_key(key) ) } }
|
39
|
+
end
|
40
|
+
|
41
|
+
def me_set( key, value ); hset( *split_key(key), value ) end
|
42
|
+
def me_setnx( key, value ); hsetnx( *split_key(key), value ) end
|
43
|
+
def me_get( key ); hget(*split_key(key)) end
|
44
|
+
|
45
|
+
def me_getset(key, value)
|
46
|
+
# multi returns array of results, also we can use raw results in case of commpression take place
|
47
|
+
# but inside pipeline, multi returns nil
|
48
|
+
ftr = []
|
49
|
+
( multi{ ftr << me_get( key ); me_set( key, value ) } || ftr )[0]
|
50
|
+
end
|
51
|
+
|
52
|
+
def me_exists?(key); hexists(*split_key(key)) end
|
53
|
+
|
54
|
+
def me_incr(key); hincrby( *split_key(key), 1 ) end
|
55
|
+
|
56
|
+
def me_incrby(key, value); hincrby(*split_key(key), value) end
|
57
|
+
|
58
|
+
# must be noticed it's not a equal replacement for a mset,
|
59
|
+
# because me_mset can be partially executed, since redis doesn't rollbacks partially failed transactions
|
60
|
+
def me_mset( *args )
|
61
|
+
#it must be multi since it keeps an order of commands
|
62
|
+
multi{ args.each_slice(2) { |key, value| me_set( key, value ) } }
|
63
|
+
end
|
64
|
+
|
65
|
+
# be aware: you cant save result of me_mget inside pipeline or multi cause pipeline returns nil
|
66
|
+
def me_mget( *keys )
|
67
|
+
pipelined { keys.each{ |key| me_get( key ) } }
|
68
|
+
end
|
69
|
+
|
70
|
+
# version to be called inside pipeline, to get values, call map(&:value)
|
71
|
+
def me_mget_p( *keys )
|
72
|
+
ftr = []
|
73
|
+
pipelined { keys.each{ |key| ftr << me_get( key ) } }
|
74
|
+
ftr
|
75
|
+
end
|
76
|
+
|
77
|
+
private
|
78
|
+
|
79
|
+
def split_key(key)
|
80
|
+
split = key.to_s.scan(/\A(.*?)(\d+)\z/).flatten
|
81
|
+
raise ArgumentError.new("Cannot split key: #{key}, key doesn't end with the numbers after zipping(#{key})!" ) if split.length == 0
|
82
|
+
|
83
|
+
split[0] = split[0] + (split[1].to_i / @hash_max_ziplist_entries).to_s
|
84
|
+
split[1] = ( split[1].to_i % @hash_max_ziplist_entries)
|
85
|
+
split[1] = split[1].to_base62 if self.class.me_config.integers_to_base62
|
86
|
+
|
87
|
+
split
|
88
|
+
end
|
89
|
+
|
90
|
+
end
|
91
|
+
|
92
|
+
end
|
@@ -0,0 +1,146 @@
|
|
1
|
+
module MeRedis
|
2
|
+
# todo warn in development that gzipped size iz bigger than strict
|
3
|
+
# use prepend for classes or extend on instances
|
4
|
+
module ZipValues
|
5
|
+
module FutureUnzip
|
6
|
+
def set_transformation(&block)
|
7
|
+
return if @transformation_set
|
8
|
+
@transformation_set = true
|
9
|
+
|
10
|
+
@old_transformation = @transformation
|
11
|
+
@transformation = -> (vl) {
|
12
|
+
if @old_transformation
|
13
|
+
@old_transformation.call( block.call(vl, self) )
|
14
|
+
else
|
15
|
+
block.call(vl, self)
|
16
|
+
end
|
17
|
+
}
|
18
|
+
self
|
19
|
+
end
|
20
|
+
|
21
|
+
# patch futures we need only when we are returning values, usual setters returns OK
|
22
|
+
COMMANDS = %i[ incr incrby hincrby get hget getset mget hgetall ].map{|cmd| [cmd, true]}.to_h
|
23
|
+
end
|
24
|
+
|
25
|
+
module PrependMethods
|
26
|
+
def pipelined( &block )
|
27
|
+
super do |redis|
|
28
|
+
block.call(redis)
|
29
|
+
_patch_futures(@client)
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
def multi( &block )
|
34
|
+
super do |redis|
|
35
|
+
block.call(redis)
|
36
|
+
_patch_futures(@client)
|
37
|
+
end
|
38
|
+
end
|
39
|
+
|
40
|
+
def _patch_futures(client)
|
41
|
+
client.futures.each do |ftr|
|
42
|
+
|
43
|
+
ftr.set_transformation do |vl|
|
44
|
+
if vl && FutureUnzip::COMMANDS[ftr._command[0]]
|
45
|
+
# we only dealing here with GET methods, so it could be hash getters or get/mget
|
46
|
+
keys = ftr._command[0][0] == 'h' ? ftr._command[1,1] : ftr._command[1..-1]
|
47
|
+
if ftr._command[0] == :mget
|
48
|
+
vl.each_with_index.map{ |v, i| zip?(keys[i]) ? self.class.get_compressor_for_key(keys[i]).decompress( v ) : v }
|
49
|
+
elsif zip?(keys[0])
|
50
|
+
compressor = self.class.get_compressor_for_key(keys[0])
|
51
|
+
# on hash commands it could be an array
|
52
|
+
vl.is_a?(Array) ? vl.map!{|v| compressor.decompress( v ) } : compressor.decompress(vl)
|
53
|
+
else
|
54
|
+
vl
|
55
|
+
end
|
56
|
+
else
|
57
|
+
vl
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
|
65
|
+
|
66
|
+
module ZlibCompressor
|
67
|
+
def self.compress(value); Zlib.deflate(value.to_s ) end
|
68
|
+
|
69
|
+
def self.decompress(value)
|
70
|
+
value ? Zlib.inflate(value) : value
|
71
|
+
rescue Zlib::DataError, Zlib::BufError
|
72
|
+
return value
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
module EmptyCompressor
|
77
|
+
def self.compress(value); value end
|
78
|
+
def self.decompress(value); value end
|
79
|
+
end
|
80
|
+
|
81
|
+
# for the global gzipping
|
82
|
+
def self.prepended(base)
|
83
|
+
base::Future.prepend(FutureUnzip)
|
84
|
+
base.prepend(PrependMethods)
|
85
|
+
|
86
|
+
base.extend( MeRedis::ClassMethods )
|
87
|
+
base.me_config.default_compressor = MeRedis::ZipValues::ZlibCompressor
|
88
|
+
end
|
89
|
+
# for object extending
|
90
|
+
def self.included(base)
|
91
|
+
base::Future.prepend(FutureUnzip)
|
92
|
+
base.prepend(PrependMethods)
|
93
|
+
|
94
|
+
base.extend( MeRedis::ClassMethods )
|
95
|
+
base.me_config.default_compressor = MeRedis::ZipValues::ZlibCompressor
|
96
|
+
end
|
97
|
+
|
98
|
+
def zip_value(value, key )
|
99
|
+
zip?(key) ? self.class.get_compressor_for_key(key).compress( value ) : value
|
100
|
+
end
|
101
|
+
|
102
|
+
def unzip_value(value, key)
|
103
|
+
return value if value.is_a?( FutureUnzip )
|
104
|
+
|
105
|
+
value.is_a?(String) && zip?(key) ? self.class.get_compressor_for_key(key).decompress( value ) : value
|
106
|
+
end
|
107
|
+
|
108
|
+
def zip?(key); self.class.zip?(key) end
|
109
|
+
|
110
|
+
# Redis prepended methods
|
111
|
+
def get( key ); unzip_value( super( key ), key) end
|
112
|
+
def set( key, value ); super( key, zip_value(value, key) ) end
|
113
|
+
|
114
|
+
def mget(*args); unzip_arr_or_future(super(*args), args ) end
|
115
|
+
def mset(*args); super( *map_msets_arr(args) ) end
|
116
|
+
|
117
|
+
def getset( key, value ); unzip_value( super( key, zip_value(value, key) ), key ) end
|
118
|
+
|
119
|
+
def hget( key, h_key ); unzip_value( super( key, h_key ), key ) end
|
120
|
+
def hset( key, h_key, value ); super( key, h_key, zip_value(value, key) ) end
|
121
|
+
def hsetnx( key, h_key, value ); super( key, h_key, zip_value(value, key) ) end
|
122
|
+
|
123
|
+
def hmset( key, *args ); super( key, map_hmsets_arr(key, *args) ) end
|
124
|
+
|
125
|
+
def hmget( key, *args ); unzip_arr_or_future( super(key, *args), key ) end
|
126
|
+
|
127
|
+
private
|
128
|
+
|
129
|
+
def unzip_arr_or_future( arr, keys )
|
130
|
+
return arr if arr.is_a?(FutureUnzip)
|
131
|
+
|
132
|
+
arr.tap { arr.each_with_index { |val, i| arr[i] = unzip_value(val,keys.is_a?(Array) ? keys[i] : keys)} }
|
133
|
+
end
|
134
|
+
|
135
|
+
def map_hmsets_arr( key, *args )
|
136
|
+
return args unless zip?(key)
|
137
|
+
counter = 0
|
138
|
+
args.map!{ |kv| (counter +=1).odd? ? kv : zip_value(kv, key ) }
|
139
|
+
end
|
140
|
+
|
141
|
+
def map_msets_arr( args )
|
142
|
+
args.tap { (args.length/2).times{ |i| args[2*i+1] = zip_value(args[2*i+1], args[2*i] ) } }
|
143
|
+
end
|
144
|
+
end
|
145
|
+
|
146
|
+
end
|
data/me-redis.gemspec
ADDED
@@ -0,0 +1,36 @@
|
|
1
|
+
# coding: utf-8
|
2
|
+
lib = File.expand_path('../lib', __FILE__)
|
3
|
+
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
+
require 'me_redis/version'
|
5
|
+
|
6
|
+
Gem::Specification.new do |spec|
|
7
|
+
spec.name = "me-redis"
|
8
|
+
spec.version = MeRedis::VERSION
|
9
|
+
spec.authors = ["alekseyl"]
|
10
|
+
spec.email = ["leshchuk@gmail.com"]
|
11
|
+
|
12
|
+
spec.summary = %q{Memory efficient redis extention}
|
13
|
+
spec.description = %q{Enable to zip keys, zip values and replace simple storage key/value pairs with hash storing}
|
14
|
+
spec.homepage = "https://github.com/alekseyl/me-redis"
|
15
|
+
spec.license = "MIT"
|
16
|
+
|
17
|
+
# Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or
|
18
|
+
# delete this section to allow pushing this gem to any host.
|
19
|
+
if spec.respond_to?(:metadata)
|
20
|
+
spec.metadata['allowed_push_host'] = "https://rubygems.org"
|
21
|
+
else
|
22
|
+
raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
|
23
|
+
end
|
24
|
+
|
25
|
+
spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
26
|
+
spec.bindir = "exe"
|
27
|
+
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
28
|
+
spec.require_paths = ["lib"]
|
29
|
+
|
30
|
+
spec.add_dependency 'redis', '>= 3.0'
|
31
|
+
spec.add_dependency 'base62-rb'
|
32
|
+
|
33
|
+
spec.add_development_dependency "bundler", "~> 1.16"
|
34
|
+
spec.add_development_dependency "rake", "~> 10.0"
|
35
|
+
spec.add_development_dependency "minitest"
|
36
|
+
end
|
metadata
ADDED
@@ -0,0 +1,137 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: me-redis
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- alekseyl
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2018-05-23 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: redis
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '3.0'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '3.0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: base62-rb
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: bundler
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '1.16'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '1.16'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: rake
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - "~>"
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '10.0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - "~>"
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '10.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: minitest
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
description: Enable to zip keys, zip values and replace simple storage key/value pairs
|
84
|
+
with hash storing
|
85
|
+
email:
|
86
|
+
- leshchuk@gmail.com
|
87
|
+
executables: []
|
88
|
+
extensions: []
|
89
|
+
extra_rdoc_files: []
|
90
|
+
files:
|
91
|
+
- ".gitignore"
|
92
|
+
- ".idea/me-redis.iml"
|
93
|
+
- ".idea/misc.xml"
|
94
|
+
- ".idea/modules.xml"
|
95
|
+
- ".idea/workspace.xml"
|
96
|
+
- ".travis.yml"
|
97
|
+
- Gemfile
|
98
|
+
- LICENSE.txt
|
99
|
+
- README.md
|
100
|
+
- Rakefile
|
101
|
+
- bin/console
|
102
|
+
- bin/setup
|
103
|
+
- lib/me_redis.rb
|
104
|
+
- lib/me_redis/hash.rb
|
105
|
+
- lib/me_redis/integer.rb
|
106
|
+
- lib/me_redis/me_redis_hot_migrator.rb
|
107
|
+
- lib/me_redis/version.rb
|
108
|
+
- lib/me_redis/zip_keys.rb
|
109
|
+
- lib/me_redis/zip_to_hash.rb
|
110
|
+
- lib/me_redis/zip_values.rb
|
111
|
+
- me-redis.gemspec
|
112
|
+
homepage: https://github.com/alekseyl/me-redis
|
113
|
+
licenses:
|
114
|
+
- MIT
|
115
|
+
metadata:
|
116
|
+
allowed_push_host: https://rubygems.org
|
117
|
+
post_install_message:
|
118
|
+
rdoc_options: []
|
119
|
+
require_paths:
|
120
|
+
- lib
|
121
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
122
|
+
requirements:
|
123
|
+
- - ">="
|
124
|
+
- !ruby/object:Gem::Version
|
125
|
+
version: '0'
|
126
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
127
|
+
requirements:
|
128
|
+
- - ">="
|
129
|
+
- !ruby/object:Gem::Version
|
130
|
+
version: '0'
|
131
|
+
requirements: []
|
132
|
+
rubyforge_project:
|
133
|
+
rubygems_version: 2.6.11
|
134
|
+
signing_key:
|
135
|
+
specification_version: 4
|
136
|
+
summary: Memory efficient redis extention
|
137
|
+
test_files: []
|