me-redis 0.1.0 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.travis.yml +5 -2
- data/CHANGELOG.md +8 -0
- data/Dockerfile +6 -0
- data/README.md +112 -49
- data/docker-compose.yml +17 -0
- data/lib/me_redis.rb +18 -19
- data/lib/me_redis/aws_config_blocker.rb +30 -0
- data/lib/me_redis/hash.rb +2 -0
- data/lib/me_redis/version.rb +1 -1
- data/lib/me_redis/zip_keys.rb +0 -1
- data/lib/me_redis/zip_to_hash.rb +2 -2
- data/lib/me_redis/zip_values.rb +44 -55
- data/me-redis.gemspec +3 -0
- metadata +35 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 42d0534ff1e6462c1ada7d35f277b4eae482285d19df0678542cc9e21f78d5f5
|
4
|
+
data.tar.gz: 56b06010bba70a0694f1ecd6d6b90147dff50af4281bc6a4993b9985ad6ccf2b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: af8c922ef45eaf8a89a39bf840e92115c21f2fca4886add3d9b4a219d5e8ce8bb989be4dd7f3b20cee3170c43978ad9d71260140092d12dd6bc0161de767c010
|
7
|
+
data.tar.gz: e5882c9e5e905a8e457668b569140d0ec678eb81eb6de0c54249e073ca2f8dc7abdc42dce10613a2aaeaa53325194f4fcaaf947ec897b7bedf43093aedaad831
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
ADDED
@@ -0,0 +1,8 @@
|
|
1
|
+
# 0.1.2
|
2
|
+
* add AwsConfigBlocker extension. Prepending MeRedis successor with this module
|
3
|
+
prevents error while using the MeRedis with AWS ElasticCache (AWS blocks all config calls).
|
4
|
+
|
5
|
+
* added docker-compose for local testing comfort
|
6
|
+
|
7
|
+
# 0.1.1
|
8
|
+
* add MockRedis extension
|
data/Dockerfile
ADDED
data/README.md
CHANGED
@@ -5,21 +5,19 @@ Me - Memory Efficient
|
|
5
5
|
This gem is delivering memory optimizations for Redis with slightest code changes.
|
6
6
|
|
7
7
|
To understand optimizations and how to use them
|
8
|
-
I suggest you to read my paper
|
8
|
+
I suggest you to read my paper: https://medium.com/@leshchuk/zipem-all-61076c7da4c
|
9
9
|
|
10
|
-
|
10
|
+
Compatibility: ruby >= 2.4, redis gem >= 3.0.4
|
11
|
+
|
12
|
+
## Features:
|
11
13
|
|
12
|
-
* seamless integration with code already in use,
|
13
|
-
add me_ prefix to some of your methods ( me_ methods implement hash memory optimization ).
|
14
|
-
It's all in MeRedis configuration, not your current code.
|
14
|
+
* seamless integration with code already in use. **It's all in MeRedis configuration, not your current code.**
|
15
15
|
|
16
|
-
* hash key/value optimization with seamless code changes,
|
17
|
-
you can replace set('object:id', value) with me_set( 'object:id', value)
|
18
|
-
and free 90 byte for each ['object:id', value] pair.
|
16
|
+
* hash key/value optimization with seamless code changes, just replace set('object:id', value) with me_set( 'object:id', value) and free upto 90 bytes for each ['object:id', value] pair.
|
19
17
|
|
20
18
|
* zips user-friendly key crumbs according to configuration, i.e. converts for example user:id to u:id
|
21
19
|
|
22
|
-
* zip integer parts of
|
20
|
+
* zip integer parts of keys with base62 encoding. Since all keys in redis are always strings, than we don't care for integer parts base, and by using base62 encoding we can shorten them upto 1.8 times shorter
|
23
21
|
|
24
22
|
* respects pipelined and multi, properly works with Futures.
|
25
23
|
|
@@ -55,7 +53,9 @@ Or install it yourself as:
|
|
55
53
|
1) less memory on redis side is better than less performance on ruby side
|
56
54
|
2) more result with less code changes,
|
57
55
|
i.e. overriding native redis methods with proper configuration basis is
|
58
|
-
preferred over mixin new methods
|
56
|
+
preferred over mixin new methods.
|
57
|
+
|
58
|
+
I wanted magic to happend: mix all kind of optimizations without making 3 call of different methods.
|
59
59
|
|
60
60
|
MeRedis based on three general optimization ideas:
|
61
61
|
* shorten keys
|
@@ -73,7 +73,7 @@ way to deal with them all is call include upon MeRedis:
|
|
73
73
|
redis = Redis.new
|
74
74
|
```
|
75
75
|
|
76
|
-
If you want to keep a clear Redis class, you can do this way:
|
76
|
+
If you want to keep a clear Redis class, you can do it this way:
|
77
77
|
|
78
78
|
```ruby
|
79
79
|
me_redis = Class.new(Redis).include(MeRedis).configure({...}).new
|
@@ -94,7 +94,22 @@ If you want to include them separately look at MeRedis.included method:
|
|
94
94
|
|
95
95
|
This is the right chain of prepending/including, so just remove unnecessary module.
|
96
96
|
|
97
|
-
###
|
97
|
+
### AWS ElasticCache usage
|
98
|
+
|
99
|
+
AWS blocks config call with exception. To prevent this behaviour prepend MeRedis successor
|
100
|
+
with AwsConfigBlocker module:
|
101
|
+
|
102
|
+
```ruby
|
103
|
+
Class.new(Redis)
|
104
|
+
.include(MeRedis)
|
105
|
+
.prepend( MeRedis::AwsConfigBlocker )
|
106
|
+
.configure({
|
107
|
+
# some config
|
108
|
+
}).new
|
109
|
+
```
|
110
|
+
|
111
|
+
|
112
|
+
### Base use
|
98
113
|
|
99
114
|
```ruby
|
100
115
|
redis = Redis.new
|
@@ -133,9 +148,9 @@ This is the right chain of prepending/including, so just remove unnecessary modu
|
|
133
148
|
|
134
149
|
As you can see you can get a result with smallest or even none code changes!
|
135
150
|
|
136
|
-
|
151
|
+
The idea is to move complexity to config.
|
137
152
|
|
138
|
-
###Config
|
153
|
+
### Config
|
139
154
|
```ruby
|
140
155
|
|
141
156
|
Redis.include(MeRedis).configure( hash_max_ziplist_entries: 512 )
|
@@ -151,9 +166,9 @@ All the ideas is to move complexity to config.
|
|
151
166
|
:hash_max_ziplist_value
|
152
167
|
|
153
168
|
# array or hash or string/sym of keys crumbs to zip,
|
154
|
-
#
|
169
|
+
# when a hash is given it used as is,
|
155
170
|
# otherwise MeRedis tries to construct hash by using first char from each key given
|
156
|
-
# + integer in base62 starting from 1 for subsequent appearence of a crumbs starting with same
|
171
|
+
# + integer in base62 starting from 1 for subsequent appearence of a crumbs starting with same char
|
157
172
|
:zip_crumbs
|
158
173
|
|
159
174
|
# set to true if you want to zip ALL integers in keys to base62 form
|
@@ -183,7 +198,7 @@ All the ideas is to move complexity to config.
|
|
183
198
|
# any kind of object which responds to compress/decompress methods
|
184
199
|
:default_compressor
|
185
200
|
```
|
186
|
-
###Config examples
|
201
|
+
### Config examples
|
187
202
|
|
188
203
|
```ruby
|
189
204
|
|
@@ -250,8 +265,8 @@ Redis.configure(
|
|
250
265
|
integers_to_base62: true,
|
251
266
|
zip_crumbs: %i[user card card_preview organization], # -> { user: :u, card: :c, card_preview: :c1, organization: :o }
|
252
267
|
compress_namespaces: {
|
253
|
-
/o:[a-zA-Z\d]+:
|
254
|
-
[:
|
268
|
+
/o:[a-zA-Z\d]+:c1/ => MeRedis::ZipValues::ZlibCompressor,
|
269
|
+
[:u, :c].map{|crumb| /o:[a-zA-Z\d]+:#{crumb}/ } => ActiveRecordJSONCompressor
|
255
270
|
}
|
256
271
|
)
|
257
272
|
|
@@ -276,20 +291,24 @@ Redis.configure(
|
|
276
291
|
compress_namespaces: %i[user card card_preview]
|
277
292
|
)
|
278
293
|
```
|
294
|
+
### Config best practices
|
279
295
|
Now I may suggest some best practices for MeRedis configure:
|
280
296
|
|
281
297
|
* explicit crumbs schema is preferable over implicit
|
282
|
-
* if you are going lazy, and use implicit schemas, than avoid keys
|
298
|
+
* if you are going lazy, and use implicit schemas, than avoid keys reshuffling,
|
283
299
|
cause it messes with your cache
|
284
|
-
* better to configure hash-max-ziplist-* in MeRedis.configure than
|
285
|
-
|
300
|
+
* better to configure hash-max-ziplist-* in MeRedis.configure than
|
301
|
+
elsewhere. And don't forget - this will reset them globally
|
302
|
+
for this Redis server!
|
303
|
+
* use MeRedis in persistent Redis-based system with extreme caution!
|
304
|
+
|
286
305
|
|
287
306
|
|
288
|
-
#Custom Compressors
|
307
|
+
# Custom Compressors
|
289
308
|
|
290
309
|
MeRedis allow you to compress values through different compressor.
|
291
310
|
Here is an example of custom compressor for ActiveRecord objects,
|
292
|
-
I use to test compression ratio against plain compression of to_json.
|
311
|
+
I use it to test compression ratio against plain compression of to_json. It takes ~40% less memory than Zlib.deflate(object.to_json).
|
293
312
|
|
294
313
|
```ruby
|
295
314
|
|
@@ -299,7 +318,7 @@ module ActiveRecordJSONCompressor
|
|
299
318
|
# like User: { first_name: 1, last_name: 2 ... },
|
300
319
|
# than your cache will be safer on schema changes.
|
301
320
|
COMPRESSOR_SCHEMAS = [User, HTag].map{|mdl|
|
302
|
-
[mdl.to_s, mdl.column_names.each_with_index.map{ |el, i| [el,
|
321
|
+
[mdl.to_s, mdl.column_names.each_with_index.map{ |el, i| [el, i.to_base62] }.to_h]
|
303
322
|
}.to_h.with_indifferent_access
|
304
323
|
|
305
324
|
REVERSE_COMPRESSOR_SCHEMA = COMPRESSOR_SCHEMAS.dup.transform_values(&:invert)
|
@@ -326,20 +345,21 @@ end
|
|
326
345
|
|
327
346
|
```
|
328
347
|
|
329
|
-
#Hot migration
|
330
|
-
MeRedis deliver additional module for hot migration to KeyZipping and ZipToHash.
|
331
|
-
We don't need one in generally for base implementation of ZipValues cause
|
332
|
-
its getter methods fallbacks to value.
|
348
|
+
# Hot migration
|
349
|
+
MeRedis deliver additional module MeRedisHotMigrator for hot migration to KeyZipping and ZipToHash.
|
333
350
|
|
334
|
-
|
335
|
-
|
336
|
-
|
351
|
+
We don't need one in generally for base implementation of ZipValues module cause
|
352
|
+
its getter methods fallbacks to values.
|
353
|
+
|
354
|
+
### Features
|
355
|
+
* mget hget hgetall get exists type getset - fallbacks to original methods for key_zipping
|
356
|
+
* me_get me_mget - fallbacks to original methods for hash zipping
|
337
357
|
* partially respects pipelining and multi
|
338
358
|
* protecting you from accidentally do many to less many migration
|
339
359
|
and from ZipToHash migration without key zipping (
|
340
360
|
though it's impossible to hot migrate from 'user:100' to 'user:1', 'B',
|
341
361
|
because of same namespace 'user' for flat key/value pair and hashes,
|
342
|
-
you'll definetely get an error )
|
362
|
+
you'll definetely get an error from Redis )
|
343
363
|
* reverse migration methods
|
344
364
|
|
345
365
|
```ruby
|
@@ -350,42 +370,59 @@ its getter methods fallbacks to value.
|
|
350
370
|
usr_1_cache = redis.me_get('user:1')
|
351
371
|
|
352
372
|
all_user_keys = redis.keys('user*')
|
373
|
+
# it doesn't delete all_user_keys!!! they still all in the Redis
|
353
374
|
redis.migrate_to_hash_representation( all_user_keys )
|
354
|
-
|
355
375
|
usr_1_cache == redis.me_get('user:1') # true
|
356
376
|
|
377
|
+
redis.del( *all_user_keys )
|
378
|
+
# now you a ready to measure memory bonus you've got
|
379
|
+
ap redis.info(:memory)
|
380
|
+
|
381
|
+
usr_1_cache == redis.me_get('user:1') # true
|
382
|
+
|
357
383
|
redis.reverse_from_hash_representation!( all_user_keys )
|
358
384
|
|
359
385
|
usr_1_cache == redis.me_get('user:1') # true
|
360
|
-
|
361
386
|
```
|
362
387
|
|
363
388
|
For persistent store use with extreme caution!!
|
364
|
-
Backup, test, test, user test and after you are sure than you may migrate.
|
389
|
+
Backup, test, test, user test and after you are completely sure than you may migrate.
|
365
390
|
|
366
|
-
Try not to stuck with
|
367
|
-
|
368
|
-
do BG deploy
|
391
|
+
Try not to stuck with MeRedisHotMigrator because it doing double amount of actions:
|
392
|
+
|
393
|
+
1. do BG deploy of code with MeRedisHotMigrator,
|
394
|
+
2. run migration in parallel,
|
395
|
+
3. replace MeRedisHotMigrator with MeRedis
|
396
|
+
4. do BG deploy
|
397
|
+
|
398
|
+
and you are done.
|
399
|
+
|
400
|
+
# Debug and broken expectations
|
401
|
+
|
402
|
+
1. Check if modules you using override appropriate method ( if not make issue :) )
|
403
|
+
2. Check the order of prepending / including modules ( look into MeRedis.included to see how things must be done )
|
404
|
+
3. Call ap Redis.me_config and check it over your expectation,
|
405
|
+
mistakes are usually in config. Don't forget - namespaces inside regexp must be in zipped form ( integers became base62 integers strings and crumbs are zipped )!
|
406
|
+
4. Check hash-max-size-value, if your average data size is greater
|
407
|
+
than hash-max-size-value than hash optimization will not bring resource savings.
|
408
|
+
5. Check hash-max-size-entries if it small than hash effect is less than expected
|
409
|
+
5. Trace through code and inspect zip_key and zip? output.
|
369
410
|
|
370
|
-
#Limitations
|
411
|
+
# Limitations
|
371
412
|
|
372
|
-
###Me_* methods limitation
|
413
|
+
### Me_* methods limitation
|
373
414
|
|
374
415
|
Some of me_methods like me_mget/me_mset/me_getset
|
375
|
-
are imitations for corresponded base methods behaviour through
|
416
|
+
are imitations for corresponded Redis base methods behaviour through
|
376
417
|
pipeline and transactions. So inside pipelined call it may not
|
377
418
|
deliver a completely equal behaviour.
|
378
419
|
|
379
|
-
me_mget has
|
420
|
+
me_mget has a method double me_mget_p in case you need to use it with futures.
|
380
421
|
|
381
|
-
###ZipKeys and ZipValues
|
422
|
+
### ZipKeys and ZipValues
|
382
423
|
As I already mention if you want to use custom prefix regex
|
383
424
|
for zipping values than it must be constructed with a crumbs substitutions,
|
384
425
|
not the original crumb, see config example.
|
385
|
-
|
386
|
-
```ruby
|
387
|
-
|
388
|
-
```
|
389
426
|
|
390
427
|
## Development
|
391
428
|
|
@@ -393,6 +430,31 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
393
430
|
|
394
431
|
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
395
432
|
|
433
|
+
## Testing
|
434
|
+
|
435
|
+
```bash
|
436
|
+
docker-compose build
|
437
|
+
docker-compose run test
|
438
|
+
```
|
439
|
+
|
440
|
+
### MockRedis
|
441
|
+
|
442
|
+
If you are using in mock_redis gem for testing purpose and see some test
|
443
|
+
breakage me-redis has a MockRedis extended class.
|
444
|
+
Its imitate single DB for all instances of MockRedis class
|
445
|
+
and add some methods MeRedis rely upon. Its not a complete replacement
|
446
|
+
for Redis class, but you can try it out.
|
447
|
+
|
448
|
+
|
449
|
+
### RedisSafety
|
450
|
+
MeRedis tests itself against RedisSafety class which is a Redis descendant class.
|
451
|
+
All instances of RedisSafety ensure that Redis DB is empty otherwise they will raise an error.
|
452
|
+
This is a precaution measure to prevent you from accidentally run MeRedis test again production DB.
|
453
|
+
|
454
|
+
```
|
455
|
+
rake test REDIS_TEST_DB=5
|
456
|
+
```
|
457
|
+
|
396
458
|
## Contributing
|
397
459
|
|
398
460
|
Bug reports and pull requests are welcome on GitHub at https://github.com/alekseyl/me-redis.
|
@@ -404,5 +466,6 @@ The gem is available as open source under the terms of the [MIT License](http://
|
|
404
466
|
|
405
467
|
## ToDo List
|
406
468
|
|
469
|
+
* add warning option for hash-max-ziplist-value overthrowing
|
407
470
|
* add keys method
|
408
|
-
* refactor readme
|
471
|
+
* refactor readme
|
data/docker-compose.yml
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
version: "3.7"
|
2
|
+
services:
|
3
|
+
redis:
|
4
|
+
image: redis
|
5
|
+
ports:
|
6
|
+
- "6379"
|
7
|
+
|
8
|
+
test:
|
9
|
+
build: .
|
10
|
+
image: me-redis-test
|
11
|
+
command: bundle exec rake test
|
12
|
+
volumes:
|
13
|
+
- '.:/me-redis'
|
14
|
+
depends_on:
|
15
|
+
- redis
|
16
|
+
environment:
|
17
|
+
REDIS_URL: redis://redis:6379
|
data/lib/me_redis.rb
CHANGED
@@ -5,7 +5,7 @@ require 'me_redis/zip_values'
|
|
5
5
|
require 'me_redis/me_redis_hot_migrator'
|
6
6
|
require 'me_redis/integer'
|
7
7
|
require 'me_redis/hash'
|
8
|
-
|
8
|
+
require 'me_redis/aws_config_blocker'
|
9
9
|
require 'zlib'
|
10
10
|
|
11
11
|
# Main ideas:
|
@@ -140,28 +140,27 @@ module MeRedis
|
|
140
140
|
end
|
141
141
|
|
142
142
|
def prepare_compressors
|
143
|
-
|
144
143
|
me_config.default_compressor ||= MeRedis::ZipValues::EmptyCompressor
|
145
144
|
|
146
145
|
me_config.compress_namespaces = case me_config.compress_namespaces
|
147
|
-
|
148
|
-
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
146
|
+
when Array
|
147
|
+
me_config.compress_namespaces.map{|ns| [replace_ns(ns), me_config.default_compressor] }.to_h
|
148
|
+
when String, Symbol, Regexp
|
149
|
+
{ replace_ns( me_config.compress_namespaces ) => me_config.default_compressor }
|
150
|
+
when Hash
|
151
|
+
me_config.compress_namespaces.inject({}) do |sum, (name_space, compressor)|
|
152
|
+
name_space.is_a?( Array ) ?
|
153
|
+
sum.merge!( name_space.map{ |ns| [replace_ns( ns), compressor] }.to_h )
|
154
|
+
: sum[replace_ns(name_space)] = compressor
|
155
|
+
sum
|
156
|
+
end
|
157
|
+
else
|
158
|
+
raise ArgumentError.new(<<~NS_ERR) if me_config.compress_namespaces
|
160
159
|
Wrong class for compress_namespaces, expected Array,
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
|
160
|
+
Hash, String or Symbol! Got: #{me_config.compress_namespaces.class.to_s}
|
161
|
+
NS_ERR
|
162
|
+
{}
|
163
|
+
end
|
165
164
|
|
166
165
|
zip_ns_finder
|
167
166
|
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
require 'ap'
|
2
|
+
module MeRedis
|
3
|
+
# how to use:
|
4
|
+
# Redis.include( MeRedis::AwsConfigBlocker )
|
5
|
+
module AwsConfigBlocker
|
6
|
+
|
7
|
+
def config( *args, &block )
|
8
|
+
print <<AWS_MSG
|
9
|
+
|
10
|
+
\e[0;33;31;1m!!!!!!!!! MeRedis AWS CONFIG BLOCKER WARNING!!!!!! \e[0;33;31;0m
|
11
|
+
You introduced AwsConfigBlocker into the ancestors chain, that means that you intend to skip Redis config call,
|
12
|
+
because AWS does not support config get/set calls by throwing an exception.
|
13
|
+
AwsConfigBlocker will block config call from reaching your Redis server!
|
14
|
+
|
15
|
+
You are calling config with arguments:
|
16
|
+
|
17
|
+
AWS_MSG
|
18
|
+
|
19
|
+
ap args
|
20
|
+
|
21
|
+
print <<END_MSG
|
22
|
+
|
23
|
+
|
24
|
+
Don't forget to setup redis param group with exact values, or you might face unexpected optimization degradation.
|
25
|
+
You can find more details in the README section.
|
26
|
+
END_MSG
|
27
|
+
{}
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/me_redis/hash.rb
CHANGED
data/lib/me_redis/version.rb
CHANGED
data/lib/me_redis/zip_keys.rb
CHANGED
data/lib/me_redis/zip_to_hash.rb
CHANGED
data/lib/me_redis/zip_values.rb
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
module MeRedis
|
2
|
-
# todo warn in development
|
3
|
-
# use prepend for classes
|
2
|
+
# todo warn in development when gzipped size iz bigger than strict
|
3
|
+
# use prepend for classes
|
4
4
|
module ZipValues
|
5
5
|
module FutureUnzip
|
6
6
|
def set_transformation(&block)
|
@@ -10,7 +10,7 @@ module MeRedis
|
|
10
10
|
@old_transformation = @transformation
|
11
11
|
@transformation = -> (vl) {
|
12
12
|
if @old_transformation
|
13
|
-
@old_transformation.call(
|
13
|
+
@old_transformation.call(block.call(vl, self))
|
14
14
|
else
|
15
15
|
block.call(vl, self)
|
16
16
|
end
|
@@ -19,50 +19,9 @@ module MeRedis
|
|
19
19
|
end
|
20
20
|
|
21
21
|
# patch futures we need only when we are returning values, usual setters returns OK
|
22
|
-
COMMANDS = %i[
|
22
|
+
COMMANDS = %i[incr incrby hincrby get hget getset mget hgetall].map{ |cmd| [cmd, true]}.to_h
|
23
23
|
end
|
24
24
|
|
25
|
-
module PrependMethods
|
26
|
-
def pipelined( &block )
|
27
|
-
super do |redis|
|
28
|
-
block.call(redis)
|
29
|
-
_patch_futures(@client)
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
|
-
def multi( &block )
|
34
|
-
super do |redis|
|
35
|
-
block.call(redis)
|
36
|
-
_patch_futures(@client)
|
37
|
-
end
|
38
|
-
end
|
39
|
-
|
40
|
-
def _patch_futures(client)
|
41
|
-
client.futures.each do |ftr|
|
42
|
-
|
43
|
-
ftr.set_transformation do |vl|
|
44
|
-
if vl && FutureUnzip::COMMANDS[ftr._command[0]]
|
45
|
-
# we only dealing here with GET methods, so it could be hash getters or get/mget
|
46
|
-
keys = ftr._command[0][0] == 'h' ? ftr._command[1,1] : ftr._command[1..-1]
|
47
|
-
if ftr._command[0] == :mget
|
48
|
-
vl.each_with_index.map{ |v, i| zip?(keys[i]) ? self.class.get_compressor_for_key(keys[i]).decompress( v ) : v }
|
49
|
-
elsif zip?(keys[0])
|
50
|
-
compressor = self.class.get_compressor_for_key(keys[0])
|
51
|
-
# on hash commands it could be an array
|
52
|
-
vl.is_a?(Array) ? vl.map!{|v| compressor.decompress( v ) } : compressor.decompress(vl)
|
53
|
-
else
|
54
|
-
vl
|
55
|
-
end
|
56
|
-
else
|
57
|
-
vl
|
58
|
-
end
|
59
|
-
end
|
60
|
-
|
61
|
-
end
|
62
|
-
end
|
63
|
-
end
|
64
|
-
|
65
|
-
|
66
25
|
module ZlibCompressor
|
67
26
|
def self.compress(value); Zlib.deflate(value.to_s ) end
|
68
27
|
|
@@ -78,23 +37,53 @@ module MeRedis
|
|
78
37
|
def self.decompress(value); value end
|
79
38
|
end
|
80
39
|
|
81
|
-
# for the global gzipping
|
82
40
|
def self.prepended(base)
|
83
41
|
base::Future.prepend(FutureUnzip)
|
84
|
-
base.prepend(PrependMethods)
|
85
42
|
|
86
|
-
base.extend(
|
87
|
-
|
43
|
+
base.extend(MeRedis::ClassMethods)
|
44
|
+
|
45
|
+
base.me_config.default_compressor = ::MeRedis::ZipValues::ZlibCompressor
|
88
46
|
end
|
89
|
-
# for object extending
|
90
|
-
def self.included(base)
|
91
|
-
base::Future.prepend(FutureUnzip)
|
92
|
-
base.prepend(PrependMethods)
|
93
47
|
|
94
|
-
|
95
|
-
|
48
|
+
def pipelined(&block)
|
49
|
+
super do |redis|
|
50
|
+
block.call(redis)
|
51
|
+
_patch_futures(@client)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def multi(&block)
|
56
|
+
super do |redis|
|
57
|
+
block.call(redis)
|
58
|
+
_patch_futures(@client)
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def _patch_futures(client)
|
63
|
+
client.futures.each do |ftr|
|
64
|
+
|
65
|
+
ftr.set_transformation do |vl|
|
66
|
+
if vl && FutureUnzip::COMMANDS[ftr._command[0]]
|
67
|
+
# we only dealing here with GET methods, so it could be hash getters or get/mget
|
68
|
+
keys = ftr._command[0][0] == 'h' ? ftr._command[1, 1] : ftr._command[1..-1]
|
69
|
+
if ftr._command[0] == :mget
|
70
|
+
vl.each_with_index.map{ |v, i| zip?(keys[i]) ? self.class.get_compressor_for_key(keys[i]).decompress( v ) : v }
|
71
|
+
elsif zip?(keys[0])
|
72
|
+
compressor = self.class.get_compressor_for_key(keys[0])
|
73
|
+
# on hash commands it could be an array
|
74
|
+
vl.is_a?(Array) ? vl.map!{ |v| compressor.decompress(v) } : compressor.decompress(vl)
|
75
|
+
else
|
76
|
+
vl
|
77
|
+
end
|
78
|
+
else
|
79
|
+
vl
|
80
|
+
end
|
81
|
+
end
|
82
|
+
|
83
|
+
end
|
96
84
|
end
|
97
85
|
|
86
|
+
|
98
87
|
def zip_value(value, key )
|
99
88
|
zip?(key) ? self.class.get_compressor_for_key(key).compress( value ) : value
|
100
89
|
end
|
data/me-redis.gemspec
CHANGED
@@ -29,8 +29,11 @@ Gem::Specification.new do |spec|
|
|
29
29
|
|
30
30
|
spec.add_dependency 'redis', '>= 3.0'
|
31
31
|
spec.add_dependency 'base62-rb'
|
32
|
+
spec.add_dependency "awesome_print"
|
32
33
|
|
33
34
|
spec.add_development_dependency "bundler", "~> 1.16"
|
34
35
|
spec.add_development_dependency "rake", "~> 10.0"
|
35
36
|
spec.add_development_dependency "minitest"
|
37
|
+
spec.add_development_dependency "pry-byebug"
|
38
|
+
|
36
39
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: me-redis
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- alekseyl
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2019-06-15 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: redis
|
@@ -38,6 +38,20 @@ dependencies:
|
|
38
38
|
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
40
|
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: awesome_print
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '0'
|
41
55
|
- !ruby/object:Gem::Dependency
|
42
56
|
name: bundler
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -80,6 +94,20 @@ dependencies:
|
|
80
94
|
- - ">="
|
81
95
|
- !ruby/object:Gem::Version
|
82
96
|
version: '0'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: pry-byebug
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
type: :development
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - ">="
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
83
111
|
description: Enable to zip keys, zip values and replace simple storage key/value pairs
|
84
112
|
with hash storing
|
85
113
|
email:
|
@@ -89,18 +117,18 @@ extensions: []
|
|
89
117
|
extra_rdoc_files: []
|
90
118
|
files:
|
91
119
|
- ".gitignore"
|
92
|
-
- ".idea/me-redis.iml"
|
93
|
-
- ".idea/misc.xml"
|
94
|
-
- ".idea/modules.xml"
|
95
|
-
- ".idea/workspace.xml"
|
96
120
|
- ".travis.yml"
|
121
|
+
- CHANGELOG.md
|
122
|
+
- Dockerfile
|
97
123
|
- Gemfile
|
98
124
|
- LICENSE.txt
|
99
125
|
- README.md
|
100
126
|
- Rakefile
|
101
127
|
- bin/console
|
102
128
|
- bin/setup
|
129
|
+
- docker-compose.yml
|
103
130
|
- lib/me_redis.rb
|
131
|
+
- lib/me_redis/aws_config_blocker.rb
|
104
132
|
- lib/me_redis/hash.rb
|
105
133
|
- lib/me_redis/integer.rb
|
106
134
|
- lib/me_redis/me_redis_hot_migrator.rb
|
@@ -130,7 +158,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
130
158
|
version: '0'
|
131
159
|
requirements: []
|
132
160
|
rubyforge_project:
|
133
|
-
rubygems_version: 2.
|
161
|
+
rubygems_version: 2.7.9
|
134
162
|
signing_key:
|
135
163
|
specification_version: 4
|
136
164
|
summary: Memory efficient redis extention
|