blind_index 1.0.0 → 2.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +36 -11
- data/LICENSE.txt +1 -1
- data/README.md +76 -50
- data/lib/blind_index.rb +23 -13
- data/lib/blind_index/backfill.rb +113 -0
- data/lib/blind_index/extensions.rb +4 -36
- data/lib/blind_index/key_generator.rb +2 -2
- data/lib/blind_index/model.rb +29 -16
- data/lib/blind_index/mongoid.rb +51 -0
- data/lib/blind_index/version.rb +1 -1
- metadata +8 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 06112a25e062c04740b0a01173300d37c364c05ef3bef0de0fa214e3dfbca61e
|
4
|
+
data.tar.gz: 41c534b7fd3266559821bba50d122a08adbf3215212304dc7c911cb62e204596
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7f3369a1b5ba9b36be21e18c3a41ea9060c779f7cb2b92c2b2539144acd5fb01d99a797b123f6f7b1a97dd53b92c6ff45ba4c5517e43bf99f3e8363875ade522
|
7
|
+
data.tar.gz: 0e5a857dd66532f72635c5ec13f56c4e864f8197533a3d300a5630cda8e9dc2d62b3c3bf771c794df2af668a2249830f6d0ba4e06a396f303131465f17dbd6a2
|
data/CHANGELOG.md
CHANGED
@@ -1,4 +1,29 @@
|
|
1
|
-
##
|
1
|
+
## 2.0.2 (2020-06-01)
|
2
|
+
|
3
|
+
- Improved error message for bad key length
|
4
|
+
- Fixed `backfill` method with relations for Mongoid
|
5
|
+
|
6
|
+
## 2.0.1 (2020-02-14)
|
7
|
+
|
8
|
+
- Added `BlindIndex.backfill` method
|
9
|
+
|
10
|
+
## 2.0.0 (2019-02-10)
|
11
|
+
|
12
|
+
- Blind indexes are updated immediately instead of in a `before_validation` callback
|
13
|
+
- Better Lockbox integration - no need to generate a separate key
|
14
|
+
- The `argon2` gem has been replaced with `argon2-kdf` for less dependencies and Windows support
|
15
|
+
- Removed deprecated `compute_email_bidx`
|
16
|
+
|
17
|
+
## 1.0.2 (2019-12-26)
|
18
|
+
|
19
|
+
- Fixed `OpenSSL::KDF` error on some platforms
|
20
|
+
- Fixed deprecation warnings in Ruby 2.7
|
21
|
+
|
22
|
+
## 1.0.1 (2019-08-16)
|
23
|
+
|
24
|
+
- Added support for Mongoid
|
25
|
+
|
26
|
+
## 1.0.0 (2019-07-08)
|
2
27
|
|
3
28
|
- Added support for master key
|
4
29
|
- Added support for Argon2id
|
@@ -11,56 +36,56 @@ Breaking changes
|
|
11
36
|
- Removed `encrypted_` prefix from columns
|
12
37
|
- Changed default encoding to Base64 strict
|
13
38
|
|
14
|
-
## 0.3.5
|
39
|
+
## 0.3.5 (2019-05-28)
|
15
40
|
|
16
41
|
- Added support for hex keys
|
17
42
|
- Added `generate_key` method
|
18
43
|
- Fixed querying with array values
|
19
44
|
|
20
|
-
## 0.3.4
|
45
|
+
## 0.3.4 (2018-12-16)
|
21
46
|
|
22
47
|
- Added `size` option
|
23
48
|
- Added sanity checks for Argon2 cost parameters
|
24
49
|
- Fixed ActiveRecord callback issues introduced in 0.3.3
|
25
50
|
|
26
|
-
## 0.3.3
|
51
|
+
## 0.3.3 (2018-11-12)
|
27
52
|
|
28
53
|
- Added support for string keys in finders
|
29
54
|
|
30
|
-
## 0.3.2
|
55
|
+
## 0.3.2 (2018-06-18)
|
31
56
|
|
32
57
|
- Added support for dynamic finders
|
33
58
|
- Added support for inherited models
|
34
59
|
|
35
|
-
## 0.3.1
|
60
|
+
## 0.3.1 (2018-06-04)
|
36
61
|
|
37
62
|
- Added scrypt and Argon2 algorithms
|
38
63
|
- Added `cost` option
|
39
64
|
|
40
|
-
## 0.3.0
|
65
|
+
## 0.3.0 (2018-06-03)
|
41
66
|
|
42
67
|
- Enforce secure key generation
|
43
68
|
- Added `encode` option
|
44
69
|
- Added `default_options` method
|
45
70
|
|
46
|
-
## 0.2.1
|
71
|
+
## 0.2.1 (2018-05-26)
|
47
72
|
|
48
73
|
- Added class method to compute blind index
|
49
74
|
- Fixed issue with cached statements
|
50
75
|
|
51
|
-
## 0.2.0
|
76
|
+
## 0.2.0 (2018-05-11)
|
52
77
|
|
53
78
|
- Added support for ActiveRecord 4.2
|
54
79
|
- Improved validation support when multiple blind indexes
|
55
80
|
- Fixed `nil` handling
|
56
81
|
|
57
|
-
## 0.1.1
|
82
|
+
## 0.1.1 (2018-04-09)
|
58
83
|
|
59
84
|
- Added support for ActiveRecord 5.2
|
60
85
|
- Added `callback` option
|
61
86
|
- Added support for `key` proc
|
62
87
|
- Fixed error inheritance
|
63
88
|
|
64
|
-
## 0.1.0
|
89
|
+
## 0.1.0 (2017-12-17)
|
65
90
|
|
66
91
|
- First release
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -2,11 +2,9 @@
|
|
2
2
|
|
3
3
|
Securely search encrypted database fields
|
4
4
|
|
5
|
-
Works with [Lockbox](https://github.com/ankane/lockbox) and [attr_encrypted](https://github.com/attr-encrypted/attr_encrypted)
|
5
|
+
Works with [Lockbox](https://github.com/ankane/lockbox) ([full example](https://ankane.org/securing-user-emails-lockbox)) and [attr_encrypted](https://github.com/attr-encrypted/attr_encrypted) ([full example](https://ankane.org/securing-user-emails-in-rails))
|
6
6
|
|
7
|
-
|
8
|
-
|
9
|
-
Check out [this post](https://ankane.org/sensitive-data-rails) for more info on securing sensitive data with Rails
|
7
|
+
Learn more about [securing sensitive data in Rails](https://ankane.org/sensitive-data-rails)
|
10
8
|
|
11
9
|
[![Build Status](https://travis-ci.org/ankane/blind_index.svg?branch=master)](https://travis-ci.org/ankane/blind_index)
|
12
10
|
|
@@ -18,7 +16,7 @@ We use [this approach](https://paragonie.com/blog/2017/05/building-searchable-en
|
|
18
16
|
|
19
17
|
An important consideration in searchable encryption is leakage, which is information an attacker can gain. Blind indexing leaks that rows have the same value. If you use this for a field like last name, an attacker can use frequency analysis to predict the values. In an active attack where an attacker can control the input values, they can learn which other values in the database match.
|
20
18
|
|
21
|
-
Here’s a [great article](https://blog.cryptographyengineering.com/2019/02/11/attack-of-the-week-searchable-encryption-and-the-ever-expanding-leakage-function/) on leakage in searchable encryption. Blind indexing has the same leakage as deterministic encryption.
|
19
|
+
Here’s a [great article](https://blog.cryptographyengineering.com/2019/02/11/attack-of-the-week-searchable-encryption-and-the-ever-expanding-leakage-function/) on leakage in searchable encryption. Blind indexing has the same leakage as [deterministic encryption](#alternatives).
|
22
20
|
|
23
21
|
## Installation
|
24
22
|
|
@@ -28,37 +26,13 @@ Add this line to your application’s Gemfile:
|
|
28
26
|
gem 'blind_index'
|
29
27
|
```
|
30
28
|
|
31
|
-
On Windows, also add:
|
32
|
-
|
33
|
-
```ruby
|
34
|
-
gem 'argon2', git: 'https://github.com/technion/ruby-argon2.git', submodules: true
|
35
|
-
```
|
36
|
-
|
37
|
-
Until `argon2 >= 2.0.1` is released.
|
38
|
-
|
39
29
|
## Getting Started
|
40
30
|
|
41
|
-
|
42
|
-
|
43
|
-
First, generate a key
|
44
|
-
|
45
|
-
```ruby
|
46
|
-
BlindIndex.generate_key
|
47
|
-
```
|
48
|
-
|
49
|
-
Store the key with your other secrets. This is typically Rails credentials or an environment variable ([dotenv](https://github.com/bkeepers/dotenv) is great for this). Be sure to use different keys in development and production. Keys don’t need to be hex-encoded, but it’s often easier to store them this way.
|
50
|
-
|
51
|
-
Set the following environment variable with your key (you can use this one in development)
|
31
|
+
Your model should already be set up with Lockbox or attr_encrypted. The examples are for a `User` model with `encrypts :email` or `attr_encrypted :email`. See the full examples for [Lockbox](https://ankane.org/securing-user-emails-lockbox) and [attr_encrypted](https://ankane.org/securing-user-emails-in-rails) if needed.
|
52
32
|
|
53
|
-
|
54
|
-
BLIND_INDEX_MASTER_KEY=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
|
55
|
-
```
|
33
|
+
Also, if you use attr_encrypted, [generate a key](#key-generation).
|
56
34
|
|
57
|
-
|
58
|
-
|
59
|
-
```ruby
|
60
|
-
BlindIndex.master_key = Rails.application.credentials.blind_index_master_key
|
61
|
-
```
|
35
|
+
---
|
62
36
|
|
63
37
|
Create a migration to add a column for the blind index
|
64
38
|
|
@@ -86,10 +60,7 @@ end
|
|
86
60
|
Backfill existing records
|
87
61
|
|
88
62
|
```ruby
|
89
|
-
|
90
|
-
user.compute_email_bidx
|
91
|
-
user.save(validate: false)
|
92
|
-
end
|
63
|
+
BlindIndex.backfill(User)
|
93
64
|
```
|
94
65
|
|
95
66
|
And query away
|
@@ -141,10 +112,7 @@ end
|
|
141
112
|
Backfill existing records
|
142
113
|
|
143
114
|
```ruby
|
144
|
-
|
145
|
-
user.compute_email_ci_bidx
|
146
|
-
user.save(validate: false)
|
147
|
-
end
|
115
|
+
BlindIndex.backfill(User, columns: [:email_ci_bidx])
|
148
116
|
```
|
149
117
|
|
150
118
|
And query away
|
@@ -171,18 +139,34 @@ You can also use virtual attributes to index data from multiple columns:
|
|
171
139
|
```ruby
|
172
140
|
class User < ApplicationRecord
|
173
141
|
attribute :initials, :string
|
142
|
+
blind_index :initials
|
174
143
|
|
175
|
-
# must come before the blind_index method so it runs first
|
176
144
|
before_validation :set_initials, if: -> { changes.key?(:first_name) || changes.key?(:last_name) }
|
177
145
|
|
178
|
-
blind_index :initials
|
179
|
-
|
180
146
|
def set_initials
|
181
147
|
self.initials = "#{first_name[0]}#{last_name[0]}"
|
182
148
|
end
|
183
149
|
end
|
184
150
|
```
|
185
151
|
|
152
|
+
## Migrating Data
|
153
|
+
|
154
|
+
If you’re encrypting a column and adding a blind index at the same time, use the `migrating` option.
|
155
|
+
|
156
|
+
```ruby
|
157
|
+
class User < ApplicationRecord
|
158
|
+
blind_index :email, migrating: true
|
159
|
+
end
|
160
|
+
```
|
161
|
+
|
162
|
+
This allows you to backfill records while still querying the unencrypted field.
|
163
|
+
|
164
|
+
```ruby
|
165
|
+
BlindIndex.backfill(User)
|
166
|
+
```
|
167
|
+
|
168
|
+
Once that completes, you can remove the `migrating` option.
|
169
|
+
|
186
170
|
## Key Rotation
|
187
171
|
|
188
172
|
To rotate keys without downtime, add a new column:
|
@@ -203,10 +187,7 @@ end
|
|
203
187
|
This will keep the new column synced going forward. Next, backfill the data:
|
204
188
|
|
205
189
|
```ruby
|
206
|
-
|
207
|
-
user.compute_rotated_email_bidx
|
208
|
-
user.save(validate: false)
|
209
|
-
end
|
190
|
+
BlindIndex.backfill(User, columns: [:email_bidx_v2])
|
210
191
|
```
|
211
192
|
|
212
193
|
Then update your model
|
@@ -252,6 +233,40 @@ test_user:
|
|
252
233
|
|
253
234
|
Be sure to include the `inspect` at the end or it won’t be encoded properly in YAML.
|
254
235
|
|
236
|
+
## Mongoid
|
237
|
+
|
238
|
+
For Mongoid, use:
|
239
|
+
|
240
|
+
```ruby
|
241
|
+
class User
|
242
|
+
field :email_bidx, type: String
|
243
|
+
end
|
244
|
+
```
|
245
|
+
|
246
|
+
## Key Generation
|
247
|
+
|
248
|
+
This is optional for Lockbox, as its master key is used by default.
|
249
|
+
|
250
|
+
Generate a key with:
|
251
|
+
|
252
|
+
```ruby
|
253
|
+
BlindIndex.generate_key
|
254
|
+
```
|
255
|
+
|
256
|
+
Store the key with your other secrets. This is typically Rails credentials or an environment variable ([dotenv](https://github.com/bkeepers/dotenv) is great for this). Be sure to use different keys in development and production. Keys don’t need to be hex-encoded, but it’s often easier to store them this way.
|
257
|
+
|
258
|
+
Set the following environment variable with your key (you can use this one in development)
|
259
|
+
|
260
|
+
```sh
|
261
|
+
BLIND_INDEX_MASTER_KEY=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
|
262
|
+
```
|
263
|
+
|
264
|
+
or create `config/initializers/blind_index.rb` with something like
|
265
|
+
|
266
|
+
```ruby
|
267
|
+
BlindIndex.master_key = Rails.application.credentials.blind_index_master_key
|
268
|
+
```
|
269
|
+
|
255
270
|
## Reference
|
256
271
|
|
257
272
|
Set default options in an initializer with:
|
@@ -286,10 +301,21 @@ end
|
|
286
301
|
|
287
302
|
## Alternatives
|
288
303
|
|
289
|
-
One alternative to blind indexing is to use a deterministic encryption scheme, like [AES-SIV](https://github.com/miscreant/miscreant). In this approach, the encrypted data will be the same for matches.
|
304
|
+
One alternative to blind indexing is to use a deterministic encryption scheme, like [AES-SIV](https://github.com/miscreant/miscreant). In this approach, the encrypted data will be the same for matches. We recommend blind indexing over deterministic encryption because:
|
305
|
+
|
306
|
+
1. You can keep encryption consistent for all fields (both searchable and non-searchable)
|
307
|
+
2. Blind indexing supports expressions
|
290
308
|
|
291
309
|
## Upgrading
|
292
310
|
|
311
|
+
### 2.0.0
|
312
|
+
|
313
|
+
2.0.0 brings a number of improvements.
|
314
|
+
|
315
|
+
- Blind indexes are updated immediately instead of in a `before_validation` callback
|
316
|
+
- Better Lockbox integration - no need to generate a separate key
|
317
|
+
- There’s a new gem for Argon2 that has no dependencies and (officially) supports Windows
|
318
|
+
|
293
319
|
### 1.0.0
|
294
320
|
|
295
321
|
1.0.0 brings a number of improvements. Here are a few to be aware of:
|
@@ -327,7 +353,7 @@ And add to your model
|
|
327
353
|
|
328
354
|
```ruby
|
329
355
|
class User < ApplicationRecord
|
330
|
-
blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"], legacy: true, rotate:
|
356
|
+
blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"], legacy: true, rotate: {}
|
331
357
|
end
|
332
358
|
```
|
333
359
|
|
@@ -408,5 +434,5 @@ To get started with development and testing:
|
|
408
434
|
git clone https://github.com/ankane/blind_index.git
|
409
435
|
cd blind_index
|
410
436
|
bundle install
|
411
|
-
rake test
|
437
|
+
bundle exec rake test
|
412
438
|
```
|
data/lib/blind_index.rb
CHANGED
@@ -1,9 +1,10 @@
|
|
1
1
|
# dependencies
|
2
2
|
require "active_support"
|
3
3
|
require "openssl"
|
4
|
-
require "argon2"
|
4
|
+
require "argon2/kdf"
|
5
5
|
|
6
6
|
# modules
|
7
|
+
require "blind_index/backfill"
|
7
8
|
require "blind_index/key_generator"
|
8
9
|
require "blind_index/model"
|
9
10
|
require "blind_index/version"
|
@@ -18,7 +19,7 @@ module BlindIndex
|
|
18
19
|
self.default_options = {}
|
19
20
|
|
20
21
|
def self.master_key
|
21
|
-
@master_key ||= ENV["BLIND_INDEX_MASTER_KEY"]
|
22
|
+
@master_key ||= ENV["BLIND_INDEX_MASTER_KEY"] || (defined?(Lockbox.master_key) && Lockbox.master_key)
|
22
23
|
end
|
23
24
|
|
24
25
|
def self.generate_bidx(value, key:, **options)
|
@@ -64,7 +65,7 @@ module BlindIndex
|
|
64
65
|
# use same bounds as rbnacl
|
65
66
|
raise BlindIndex::Error, "m must be between 3 and 22" if m < 3 || m > 22
|
66
67
|
|
67
|
-
|
68
|
+
Argon2::KDF.argon2id(value, salt: key, t: t, m: m, p: 1, length: size)
|
68
69
|
when :pbkdf2_sha256
|
69
70
|
iterations = cost_options[:iterations] || options[:iterations] || (options[:slow] ? 100000 : 10000)
|
70
71
|
OpenSSL::PKCS5.pbkdf2_hmac(value, key, iterations, size, "sha256")
|
@@ -78,7 +79,7 @@ module BlindIndex
|
|
78
79
|
# use same bounds as rbnacl
|
79
80
|
raise BlindIndex::Error, "m must be between 3 and 22" if m < 3 || m > 22
|
80
81
|
|
81
|
-
|
82
|
+
Argon2::KDF.argon2i(value, salt: key, t: t, m: m, p: 1, length: size)
|
82
83
|
when :scrypt
|
83
84
|
n = cost_options[:n] || 4096
|
84
85
|
r = cost_options[:r] || 8
|
@@ -116,32 +117,41 @@ module BlindIndex
|
|
116
117
|
key
|
117
118
|
end
|
118
119
|
|
119
|
-
def self.decode_key(key)
|
120
|
+
def self.decode_key(key, name: "Key")
|
120
121
|
# decode hex key
|
121
122
|
if key.encoding != Encoding::BINARY && key =~ /\A[0-9a-f]{64}\z/i
|
122
123
|
key = [key].pack("H*")
|
123
124
|
end
|
124
125
|
|
125
|
-
raise BlindIndex::Error, "
|
126
|
-
raise BlindIndex::Error, "
|
126
|
+
raise BlindIndex::Error, "#{name} must be 32 bytes (64 hex digits)" if key.bytesize != 32
|
127
|
+
raise BlindIndex::Error, "#{name} must use binary encoding" if key.encoding != Encoding::BINARY
|
127
128
|
|
128
129
|
key
|
129
130
|
end
|
131
|
+
|
132
|
+
def self.backfill(relation, columns: nil, batch_size: 1000)
|
133
|
+
Backfill.new(relation, columns: columns, batch_size: batch_size).perform
|
134
|
+
end
|
130
135
|
end
|
131
136
|
|
132
137
|
ActiveSupport.on_load(:active_record) do
|
133
138
|
require "blind_index/extensions"
|
134
139
|
extend BlindIndex::Model
|
135
140
|
|
136
|
-
|
137
|
-
ActiveRecord::TableMetadata.prepend(BlindIndex::Extensions::TableMetadata)
|
138
|
-
else
|
139
|
-
ActiveRecord::PredicateBuilder.singleton_class.prepend(BlindIndex::Extensions::PredicateBuilder)
|
140
|
-
end
|
141
|
-
|
141
|
+
ActiveRecord::TableMetadata.prepend(BlindIndex::Extensions::TableMetadata)
|
142
142
|
ActiveRecord::DynamicMatchers::Method.prepend(BlindIndex::Extensions::DynamicMatchers)
|
143
143
|
|
144
144
|
unless ActiveRecord::VERSION::STRING.start_with?("5.1.")
|
145
145
|
ActiveRecord::Validations::UniquenessValidator.prepend(BlindIndex::Extensions::UniquenessValidator)
|
146
146
|
end
|
147
147
|
end
|
148
|
+
|
149
|
+
if defined?(Mongoid)
|
150
|
+
# TODO find better ActiveModel hook
|
151
|
+
require "active_model/callbacks"
|
152
|
+
ActiveModel::Callbacks.include(BlindIndex::Model)
|
153
|
+
|
154
|
+
require "blind_index/mongoid"
|
155
|
+
Mongoid::Criteria.prepend(BlindIndex::Mongoid::Criteria)
|
156
|
+
Mongoid::Validatable::UniquenessValidator.prepend(BlindIndex::Mongoid::UniquenessValidator)
|
157
|
+
end
|
@@ -0,0 +1,113 @@
|
|
1
|
+
module BlindIndex
|
2
|
+
class Backfill
|
3
|
+
attr_reader :blind_indexes
|
4
|
+
|
5
|
+
def initialize(relation, batch_size:, columns:)
|
6
|
+
@relation = relation
|
7
|
+
@transaction = @relation.respond_to?(:transaction)
|
8
|
+
@batch_size = batch_size
|
9
|
+
@blind_indexes = @relation.blind_indexes
|
10
|
+
filter_columns!(columns) if columns
|
11
|
+
end
|
12
|
+
|
13
|
+
def perform
|
14
|
+
each_batch do |records|
|
15
|
+
backfill_records(records)
|
16
|
+
end
|
17
|
+
end
|
18
|
+
|
19
|
+
private
|
20
|
+
|
21
|
+
# modify in-place
|
22
|
+
def filter_columns!(columns)
|
23
|
+
columns = Array(columns).map(&:to_s)
|
24
|
+
blind_indexes.select! { |_, v| columns.include?(v[:bidx_attribute]) }
|
25
|
+
bad_columns = columns - blind_indexes.map { |_, v| v[:bidx_attribute] }
|
26
|
+
raise ArgumentError, "Bad column: #{bad_columns.first}" if bad_columns.any?
|
27
|
+
end
|
28
|
+
|
29
|
+
def build_relation
|
30
|
+
# build relation
|
31
|
+
relation = @relation
|
32
|
+
|
33
|
+
if defined?(ActiveRecord::Base) && relation.is_a?(ActiveRecord::Base)
|
34
|
+
relation = relation.unscoped
|
35
|
+
end
|
36
|
+
|
37
|
+
# convert from possible class to ActiveRecord::Relation or Mongoid::Criteria
|
38
|
+
relation = relation.all
|
39
|
+
|
40
|
+
attributes = blind_indexes.map { |_, v| v[:bidx_attribute] }
|
41
|
+
|
42
|
+
if defined?(ActiveRecord::Relation) && relation.is_a?(ActiveRecord::Relation)
|
43
|
+
base_relation = relation.unscoped
|
44
|
+
or_relation = relation.unscoped
|
45
|
+
|
46
|
+
attributes.each_with_index do |attribute, i|
|
47
|
+
or_relation =
|
48
|
+
if i == 0
|
49
|
+
base_relation.where(attribute => nil)
|
50
|
+
else
|
51
|
+
or_relation.or(base_relation.where(attribute => nil))
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
relation.merge(or_relation)
|
56
|
+
else
|
57
|
+
relation.merge(relation.unscoped.or(attributes.map { |a| {a => nil} }))
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
def each_batch
|
62
|
+
relation = build_relation
|
63
|
+
|
64
|
+
if relation.respond_to?(:find_in_batches)
|
65
|
+
relation.find_in_batches(batch_size: @batch_size) do |records|
|
66
|
+
yield records
|
67
|
+
end
|
68
|
+
else
|
69
|
+
# https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
|
70
|
+
# use cursor for Mongoid
|
71
|
+
records = []
|
72
|
+
relation.all.each do |record|
|
73
|
+
records << record
|
74
|
+
if records.length == @batch_size
|
75
|
+
yield records
|
76
|
+
records = []
|
77
|
+
end
|
78
|
+
end
|
79
|
+
yield records if records.any?
|
80
|
+
end
|
81
|
+
end
|
82
|
+
|
83
|
+
def backfill_records(records)
|
84
|
+
# do expensive blind index computation outside of transaction
|
85
|
+
records.each do |record|
|
86
|
+
blind_indexes.each do |k, v|
|
87
|
+
record.send("compute_#{k}_bidx") if !record.send(v[:bidx_attribute])
|
88
|
+
end
|
89
|
+
end
|
90
|
+
|
91
|
+
# don't need to save records that went from nil => nil
|
92
|
+
records.select! { |r| r.changed? }
|
93
|
+
|
94
|
+
if records.any?
|
95
|
+
with_transaction do
|
96
|
+
records.each do |record|
|
97
|
+
record.save!(validate: false)
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
end
|
102
|
+
|
103
|
+
def with_transaction
|
104
|
+
if @transaction
|
105
|
+
@relation.transaction do
|
106
|
+
yield
|
107
|
+
end
|
108
|
+
else
|
109
|
+
yield
|
110
|
+
end
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
@@ -1,6 +1,5 @@
|
|
1
1
|
module BlindIndex
|
2
2
|
module Extensions
|
3
|
-
# ActiveRecord 5.0+
|
4
3
|
module TableMetadata
|
5
4
|
def resolve_column_aliases(hash)
|
6
5
|
new_hash = super
|
@@ -10,9 +9,9 @@ module BlindIndex
|
|
10
9
|
value = new_hash.delete(key)
|
11
10
|
new_hash[bi[:bidx_attribute]] =
|
12
11
|
if value.is_a?(Array)
|
13
|
-
value.map { |v| BlindIndex.generate_bidx(v, bi) }
|
12
|
+
value.map { |v| BlindIndex.generate_bidx(v, **bi) }
|
14
13
|
else
|
15
|
-
BlindIndex.generate_bidx(value, bi)
|
14
|
+
BlindIndex.generate_bidx(value, **bi)
|
16
15
|
end
|
17
16
|
end
|
18
17
|
end
|
@@ -29,42 +28,11 @@ module BlindIndex
|
|
29
28
|
end
|
30
29
|
end
|
31
30
|
|
32
|
-
# ActiveRecord 4.2
|
33
|
-
module PredicateBuilder
|
34
|
-
def resolve_column_aliases(klass, hash)
|
35
|
-
new_hash = super
|
36
|
-
if has_blind_indexes?(klass)
|
37
|
-
hash.each do |key, _|
|
38
|
-
if key.respond_to?(:to_sym) && (bi = klass.blind_indexes[key.to_sym]) && !new_hash[key].is_a?(ActiveRecord::StatementCache::Substitute)
|
39
|
-
value = new_hash.delete(key)
|
40
|
-
new_hash[bi[:bidx_attribute]] =
|
41
|
-
if value.is_a?(Array)
|
42
|
-
value.map { |v| BlindIndex.generate_bidx(v, bi) }
|
43
|
-
else
|
44
|
-
BlindIndex.generate_bidx(value, bi)
|
45
|
-
end
|
46
|
-
end
|
47
|
-
end
|
48
|
-
end
|
49
|
-
new_hash
|
50
|
-
end
|
51
|
-
|
52
|
-
@@blind_index_cache = {}
|
53
|
-
|
54
|
-
# memoize for performance
|
55
|
-
def has_blind_indexes?(klass)
|
56
|
-
if @@blind_index_cache[klass].nil?
|
57
|
-
@@blind_index_cache[klass] = klass.respond_to?(:blind_indexes)
|
58
|
-
end
|
59
|
-
@@blind_index_cache[klass]
|
60
|
-
end
|
61
|
-
end
|
62
|
-
|
63
31
|
module UniquenessValidator
|
64
32
|
if ActiveRecord::VERSION::STRING >= "5.2"
|
65
33
|
def build_relation(klass, attribute, value)
|
66
34
|
if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
|
67
|
-
value = BlindIndex.generate_bidx(value, bi)
|
35
|
+
value = BlindIndex.generate_bidx(value, **bi)
|
68
36
|
attribute = bi[:bidx_attribute]
|
69
37
|
end
|
70
38
|
super(klass, attribute, value)
|
@@ -72,7 +40,7 @@ module BlindIndex
|
|
72
40
|
else
|
73
41
|
def build_relation(klass, table, attribute, value)
|
74
42
|
if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
|
75
|
-
value = BlindIndex.generate_bidx(value, bi)
|
43
|
+
value = BlindIndex.generate_bidx(value, **bi)
|
76
44
|
attribute = bi[:bidx_attribute]
|
77
45
|
end
|
78
46
|
super(klass, table, attribute, value)
|
@@ -11,7 +11,7 @@ module BlindIndex
|
|
11
11
|
raise ArgumentError, "Missing field for key generation" if bidx_attribute.to_s.empty?
|
12
12
|
|
13
13
|
c = "\x7E"*32
|
14
|
-
root_key = hkdf(BlindIndex.decode_key(@master_key), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
|
14
|
+
root_key = hkdf(BlindIndex.decode_key(@master_key, name: "Master key"), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
|
15
15
|
hash_hmac("sha256", pack([table, bidx_attribute, bidx_attribute]), root_key)
|
16
16
|
end
|
17
17
|
|
@@ -22,7 +22,7 @@ module BlindIndex
|
|
22
22
|
end
|
23
23
|
|
24
24
|
def hkdf(ikm, salt:, info:, length:, hash:)
|
25
|
-
if OpenSSL::KDF.
|
25
|
+
if defined?(OpenSSL::KDF.hkdf)
|
26
26
|
return OpenSSL::KDF.hkdf(ikm, salt: salt, info: info, length: length, hash: hash)
|
27
27
|
end
|
28
28
|
|
data/lib/blind_index/model.rb
CHANGED
@@ -33,7 +33,7 @@ module BlindIndex
|
|
33
33
|
class_method_name = :"generate_#{name}_bidx"
|
34
34
|
|
35
35
|
key = options[:key]
|
36
|
-
key ||= -> { BlindIndex.index_key(table: table_name, bidx_attribute: bidx_attribute, master_key: options[:master_key], encode: false) }
|
36
|
+
key ||= -> { BlindIndex.index_key(table: try(:table_name) || collection_name.to_s, bidx_attribute: bidx_attribute, master_key: options[:master_key], encode: false) }
|
37
37
|
|
38
38
|
class_eval do
|
39
39
|
@blind_indexes ||= {}
|
@@ -61,20 +61,33 @@ module BlindIndex
|
|
61
61
|
)
|
62
62
|
|
63
63
|
define_singleton_method class_method_name do |value|
|
64
|
-
BlindIndex.generate_bidx(value, blind_indexes[name])
|
65
|
-
end
|
66
|
-
|
67
|
-
define_singleton_method method_name do |value|
|
68
|
-
ActiveSupport::Deprecation.warn("Use #{class_method_name} instead")
|
69
|
-
send(class_method_name, value)
|
64
|
+
BlindIndex.generate_bidx(value, **blind_indexes[name])
|
70
65
|
end
|
71
66
|
|
72
67
|
define_method method_name do
|
73
|
-
|
68
|
+
send("#{bidx_attribute}=", self.class.send(class_method_name, send(attribute)))
|
74
69
|
end
|
75
70
|
|
76
71
|
if callback
|
77
|
-
|
72
|
+
activerecord = defined?(ActiveRecord) && self < ActiveRecord::Base
|
73
|
+
|
74
|
+
# TODO reuse module
|
75
|
+
m = Module.new do
|
76
|
+
define_method "#{attribute}=" do |value|
|
77
|
+
result = super(value)
|
78
|
+
send(method_name)
|
79
|
+
result
|
80
|
+
end
|
81
|
+
|
82
|
+
unless activerecord
|
83
|
+
define_method "reset_#{attribute}!" do
|
84
|
+
result = super()
|
85
|
+
send(method_name)
|
86
|
+
result
|
87
|
+
end
|
88
|
+
end
|
89
|
+
end
|
90
|
+
prepend m
|
78
91
|
end
|
79
92
|
|
80
93
|
# use include so user can override
|
@@ -82,14 +95,14 @@ module BlindIndex
|
|
82
95
|
end
|
83
96
|
end
|
84
97
|
end
|
85
|
-
end
|
86
98
|
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
99
|
+
module InstanceMethods
|
100
|
+
def read_attribute_for_validation(key)
|
101
|
+
if (bi = self.class.blind_indexes[key])
|
102
|
+
send(bi[:attribute])
|
103
|
+
else
|
104
|
+
super
|
105
|
+
end
|
93
106
|
end
|
94
107
|
end
|
95
108
|
end
|
@@ -0,0 +1,51 @@
|
|
1
|
+
module BlindIndex
|
2
|
+
module Mongoid
|
3
|
+
module Criteria
|
4
|
+
private
|
5
|
+
|
6
|
+
def expr_query(criterion)
|
7
|
+
if criterion.is_a?(Hash) && klass.respond_to?(:blind_indexes)
|
8
|
+
criterion.keys.each do |key|
|
9
|
+
key_sym = (key.is_a?(::Mongoid::Criteria::Queryable::Key) ? key.name : key).to_sym
|
10
|
+
|
11
|
+
if (bi = klass.blind_indexes[key_sym])
|
12
|
+
value = criterion.delete(key)
|
13
|
+
|
14
|
+
bidx_key =
|
15
|
+
if key.is_a?(::Mongoid::Criteria::Queryable::Key)
|
16
|
+
::Mongoid::Criteria::Queryable::Key.new(
|
17
|
+
bi[:bidx_attribute],
|
18
|
+
key.strategy,
|
19
|
+
key.operator,
|
20
|
+
key.expanded,
|
21
|
+
&key.block
|
22
|
+
)
|
23
|
+
else
|
24
|
+
bi[:bidx_attribute]
|
25
|
+
end
|
26
|
+
|
27
|
+
criterion[bidx_key] =
|
28
|
+
if value.is_a?(Array)
|
29
|
+
value.map { |v| BlindIndex.generate_bidx(v, bi) }
|
30
|
+
else
|
31
|
+
BlindIndex.generate_bidx(value, bi)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
super(criterion)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
module UniquenessValidator
|
42
|
+
def create_criteria(base, document, attribute, value)
|
43
|
+
if base.respond_to?(:blind_indexes) && (bi = base.blind_indexes[attribute])
|
44
|
+
value = BlindIndex.generate_bidx(value, bi)
|
45
|
+
attribute = bi[:bidx_attribute]
|
46
|
+
end
|
47
|
+
super(base, document, attribute, value)
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
data/lib/blind_index/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: blind_index
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-06-01 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
@@ -25,19 +25,19 @@ dependencies:
|
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: '5'
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
|
-
name: argon2
|
28
|
+
name: argon2-kdf
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
31
|
- - ">="
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version:
|
33
|
+
version: 0.1.1
|
34
34
|
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
38
|
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version:
|
40
|
+
version: 0.1.1
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: bundler
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
@@ -174,9 +174,11 @@ files:
|
|
174
174
|
- LICENSE.txt
|
175
175
|
- README.md
|
176
176
|
- lib/blind_index.rb
|
177
|
+
- lib/blind_index/backfill.rb
|
177
178
|
- lib/blind_index/extensions.rb
|
178
179
|
- lib/blind_index/key_generator.rb
|
179
180
|
- lib/blind_index/model.rb
|
181
|
+
- lib/blind_index/mongoid.rb
|
180
182
|
- lib/blind_index/version.rb
|
181
183
|
homepage: https://github.com/ankane/blind_index
|
182
184
|
licenses:
|
@@ -197,7 +199,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
197
199
|
- !ruby/object:Gem::Version
|
198
200
|
version: '0'
|
199
201
|
requirements: []
|
200
|
-
rubygems_version: 3.
|
202
|
+
rubygems_version: 3.1.2
|
201
203
|
signing_key:
|
202
204
|
specification_version: 4
|
203
205
|
summary: Securely search encrypted database fields
|