sequel 5.42.0 → 5.43.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 41e987d42df8d1b3ab41dd6555b5e338b94ba0e2fcd11fa3db7f9b955920d533
4
- data.tar.gz: fdcaed20caa20bbaffe5854cbcd6836787e92a2bbadb89373337602926d476ca
3
+ metadata.gz: 69c695b559f3d5c19284905e8bad5a8775cced221f5cd3f5bb1d89daa9c0785c
4
+ data.tar.gz: 03cd2d01139038c3e6d1e1232f6b48a104657391aeefd82feab2636044480eba
5
5
  SHA512:
6
- metadata.gz: c9a3927fe546ee7f91497e5bd50e9fa9241dd46aa7d3b3c331d10f8ac369eedd204a488207c509e909d9eb5d7deeeb13ab20888568f7e91f53fba526323df566
7
- data.tar.gz: a9108c8ce0d78d93c94f0a947171687022dd20ee1305633533340f1cb1a71d012248f9a9bb72eb7af28cc258240ec2e8f090683d1ad7a68c33f69d261c684ea0
6
+ metadata.gz: 4b6f0b096657603c11cf54b1b15b660c5b85a697e8b31b69f97f33d579401a73915a315b771467e56159313243c5643bb3ca5fd8c208f2c443d8c7536cba9fc0
7
+ data.tar.gz: 1a44276ab427bc2f9a82e2f651950100360df998af75a9d3ca08618acfa3778b02764f544738569c5947b350ac8b2485b825052869ac882728707014e90633a8
data/CHANGELOG CHANGED
@@ -1,3 +1,7 @@
1
+ === 5.43.0 (2021-04-01)
2
+
3
+ * Add column_encryption plugin, for encrypting column values (jeremyevans)
4
+
1
5
  === 5.42.0 (2021-03-01)
2
6
 
3
7
  * Make the ado timestamp conversion proc a normal conversion proc that can be overridden similar to other conversion procs (jeremyevans)
@@ -0,0 +1,98 @@
1
+ = New Features
2
+
3
+ * A column_encryption plugin has been added to support encrypting the
4
+ content of individual columns in a table.
5
+
6
+ Column values are encrypted with AES-256-GCM using a per-value
7
+ cipher key derived from a key provided in the configuration using
8
+ HMAC-SHA256.
9
+
10
+ If you would like to support encryption of columns in more than one
11
+ model, you should probably load the plugin into the parent class of
12
+ your models and specify the keys:
13
+
14
+ Sequel::Model.plugin :column_encryption do |enc|
15
+ enc.key 0, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
16
+ end
17
+
18
+ This specifies a single master encryption key. Unless you are
19
+ actively rotating keys, it is best to use a single master key.
20
+
21
+ In the above call, 0 is the id of the key, and
22
+ ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"] is the content of the key, which
23
+ must be a string with exactly 32 bytes. As indicated, this key
24
+ should not be hardcoded or otherwise committed to the source control
25
+ repository.
26
+
27
+ For models that need encrypted columns, you load the plugin again,
28
+ but specify the columns to encrypt:
29
+
30
+ ConfidentialModel.plugin :column_encryption do |enc|
31
+ enc.column :encrypted_column_name
32
+ enc.column :searchable_column_name, searchable: true
33
+ enc.column :ci_searchable_column_name, searchable: :case_insensitive
34
+ end
35
+
36
+ With this, all three specified columns (encrypted_column_name,
37
+ searchable_column_name, and ci_searchable_column_name) will be
38
+ marked as encrypted columns. When you run the following code:
39
+
40
+ ConfidentialModel.create(
41
+ encrypted_column_name: 'These',
42
+ searchable_column_name: 'will be',
43
+ ci_searchable_column_name: 'Encrypted'
44
+ )
45
+
46
+ It will save encrypted versions to the database.
47
+ encrypted_column_name will not be searchable, searchable_column_name
48
+ will be searchable with an exact match, and
49
+ ci_searchable_column_name will be searchable with a case insensitive
50
+ match.
51
+
52
+ To search searchable encrypted columns, use with_encrypted_value.
53
+ This example code will return the model instance created in the code
54
+ example in the previous section:
55
+
56
+ ConfidentialModel.
57
+ with_encrypted_value(:searchable_column_name, "will be")
58
+ with_encrypted_value(:ci_searchable_column_name, "encrypted").
59
+ first
60
+
61
+ To rotate encryption keys, add a new key above the existing key,
62
+ with a new key ID:
63
+
64
+ Sequel::Model.plugin :column_encryption do |enc|
65
+ enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
66
+ enc.key 0, ENV["SEQUEL_OLD_COLUMN_ENCRYPTION_KEY"]
67
+ end
68
+
69
+ Newly encrypted data will then use the new key. Records encrypted
70
+ with the older key will still be decrypted correctly.
71
+
72
+ To force reencryption for existing records that are using the older
73
+ key, you can use the needing_reencryption dataset method and the
74
+ reencrypt instance method. For a small number of records, you can
75
+ probably do:
76
+
77
+ ConfidentialModel.needing_reencryption.all(&:reencrypt)
78
+
79
+ With more than a small number of records, you'll want to do this in
80
+ batches. It's possible you could use an approach such as:
81
+
82
+ ds = ConfidentialModel.needing_reencryption.limit(100)
83
+ true until ds.all(&:reencrypt).empty?
84
+
85
+ After all values have been reencrypted for all models, and no models
86
+ use the older encryption key, you can remove it from the
87
+ configuration:
88
+
89
+ Sequel::Model.plugin :column_encryption do |enc|
90
+ enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
91
+ end
92
+
93
+ The column_encryption plugin supports encrypting serialized data,
94
+ as well as enforcing uniquenss of searchable encrypted columns
95
+ (in the absence of key rotation). By design, it does not support
96
+ compression, mixing encrypted and unencrypted data in the same
97
+ column, or support arbitrary encryption ciphers. See the plugin
98
+ documentation for more details.
@@ -54,7 +54,6 @@ module Sequel
54
54
  end
55
55
  parts = {}
56
56
  interval.each{|k,v| parts[k] = -v unless v.nil?}
57
- parts
58
57
  DateAdd.new(expr, parts, opts)
59
58
  end
60
59
  end
@@ -0,0 +1,711 @@
1
+ # frozen-string-literal: true
2
+
3
+ # :nocov:
4
+ raise(Sequel::Error, "Sequel column_encryption plugin requires ruby 2.3 or greater") unless RUBY_VERSION >= '2.3'
5
+ # :nocov:
6
+
7
+ require 'openssl'
8
+
9
+ begin
10
+ OpenSSL::Cipher.new("aes-256-gcm")
11
+ rescue OpenSSL::Cipher::CipherError
12
+ # :nocov:
13
+ raise LoadError, "Sequel column_encryption plugin requires the aes-256-gcm cipher"
14
+ # :nocov:
15
+ end
16
+
17
+ require 'base64'
18
+ require 'securerandom'
19
+
20
+ module Sequel
21
+ module Plugins
22
+ # The column_encryption plugin adds support for encrypting the content of individual
23
+ # columns in a table.
24
+ #
25
+ # Column values are encrypted with AES-256-GCM using a per-value cipher key derived from
26
+ # a key provided in the configuration using HMAC-SHA256.
27
+ #
28
+ # = Usage
29
+ #
30
+ # If you would like to support encryption of columns in more than one model, you should
31
+ # probably load the plugin into the parent class of your models and specify the keys:
32
+ #
33
+ # Sequel::Model.plugin :column_encryption do |enc|
34
+ # enc.key 0, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
35
+ # end
36
+ #
37
+ # This specifies a single master encryption key. Unless you are actively rotating keys,
38
+ # it is best to use a single master key. Rotation of encryption keys will be discussed
39
+ # in a later section.
40
+ #
41
+ # In the above call, <tt>0</tt> is the id of the key, and the
42
+ # <tt>ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]</tt> is the content of the key, which must be
43
+ # a string with exactly 32 bytes. As indicated, this key should not be hardcoded or
44
+ # otherwise committed to the source control repository.
45
+ #
46
+ # For models that need encrypted columns, you load the plugin again, but specify the
47
+ # columns to encrypt:
48
+ #
49
+ # ConfidentialModel.plugin :column_encryption do |enc|
50
+ # enc.column :encrypted_column_name
51
+ # enc.column :searchable_column_name, searchable: true
52
+ # enc.column :ci_searchable_column_name, searchable: :case_insensitive
53
+ # end
54
+ #
55
+ # With this, all three specified columns (+encrypted_column_name+, +searchable_column_name+,
56
+ # and +ci_searchable_column_name+) will be marked as encrypted columns. When you run the
57
+ # following code:
58
+ #
59
+ # ConfidentialModel.create(
60
+ # encrypted_column_name: 'These',
61
+ # searchable_column_name: 'will be',
62
+ # ci_searchable_column_name: 'Encrypted'
63
+ # )
64
+ #
65
+ # It will save encrypted versions to the database. +encrypted_column_name+ will not be
66
+ # searchable, +searchable_column_name+ will be searchable with an exact match, and
67
+ # +ci_searchable_column_name+ will be searchable with a case insensitive match. See section
68
+ # below for details on searching.
69
+ #
70
+ # It is possible to have model-specific keys by specifying both the +key+ and +column+ methods
71
+ # in the model:
72
+ #
73
+ # ConfidentialModel.plugin :column_encryption do |enc|
74
+ # enc.key 0, ENV["SEQUEL_MODEL_SPECIFIC_ENCRYPTION_KEY"]
75
+ #
76
+ # enc.column :encrypted_column_name
77
+ # enc.column :searchable_column_name, searchable: true
78
+ # enc.column :ci_searchable_column_name, searchable: :case_insensitive
79
+ # end
80
+ #
81
+ # When the +key+ method is called inside the plugin block, previous keys are ignored,
82
+ # and only the new keys specified will be used. This approach would allow the
83
+ # +ConfidentialModel+ to use the model specific encryption keys, and other models
84
+ # to use the default keys specified in the parent class.
85
+ #
86
+ # The +key+ and +column+ methods inside the plugin block support additional options.
87
+ # The +key+ method supports the following options:
88
+ #
89
+ # :auth_data :: The authentication data to use for the AES-256-GCM cipher. Defaults
90
+ # to the empty string.
91
+ # :padding :: The number of padding bytes to use. For security, data is padded so that
92
+ # a database administrator cannot determine the exact size of the
93
+ # unencrypted data. By default, this value is 8, which means that
94
+ # unencrypted data will be padded to a multiple of 8 bytes. Up to twice as
95
+ # much padding as specified will be used, as the number of padding bytes
96
+ # is partially randomized.
97
+ #
98
+ # The +column+ method supports the following options:
99
+ #
100
+ # :searchable :: Whether the column is searchable. This should not be used unless
101
+ # searchability is needed, as it can allow the database administrator
102
+ # to determine whether two distinct rows have the same unencrypted
103
+ # data (but not what that data is). This can be set to +true+ to allow
104
+ # searching with an exact match, or +:case_insensitive+ for a case
105
+ # insensitive match.
106
+ # :search_both :: This should only be used if you have previously switched the
107
+ # +:searchable+ option from +true+ to +:case_insensitive+ or vice-versa,
108
+ # and would like the search to return values that have not yet been
109
+ # reencrypted. Note that switching from +true+ to +:case_insensitive+
110
+ # isn't a problem, but switching from +:case_insensitive+ to +true+ and
111
+ # using this option can cause the search to return values that are
112
+ # not an exact match. You should manually filter those objects
113
+ # after decrypting if you want to ensure an exact match.
114
+ # :format :: The format of the column, if you want to perform serialization before
115
+ # encryption and deserialization after decryption. Can be either a
116
+ # symbol registered with the serialization plugin or an array of two
117
+ # callables, the first for serialization and the second for deserialization.
118
+ #
119
+ # The +column+ method also supports a block for column-specific keys:
120
+ #
121
+ # ConfidentialModel.plugin :column_encryption do |enc|
122
+ # enc.column :encrypted_column_name do |cenc|
123
+ # cenc.key 0, ENV["SEQUEL_COLUMN_SPECIFIC_ENCRYPTION_KEY"]
124
+ # end
125
+ #
126
+ # enc.column :searchable_column_name, searchable: true
127
+ # enc.column :ci_searchable_column_name, searchable: :case_insensitive
128
+ # end
129
+ #
130
+ # In this case, the <tt>ENV["SEQUEL_COLUMN_SPECIFIC_ENCRYPTION_KEY"]</tt> key will
131
+ # only be used for the +:encrypted_column_name+ column, and not the other columns.
132
+ #
133
+ # Note that there isn't a security reason to prefer either model-specific or
134
+ # column-specific keys, as the actual cipher key used is unique per column value.
135
+ #
136
+ # Note that changing the key_id, key string, or auth_data for an existing key will
137
+ # break decryption of values encrypted with that key. If you would like to change
138
+ # any aspect of the key, add a new key, rotate to the new encryption key, and then
139
+ # remove the previous key, as described in the section below on key rotation.
140
+ #
141
+ # = Searching Encrypted Values
142
+ #
143
+ # To search searchable encrypted columns, use +with_encrypted_value+. This example
144
+ # code will return the model instance created in the code example in the previous
145
+ # section:
146
+ #
147
+ # ConfidentialModel.
148
+ # with_encrypted_value(:searchable_column_name, "will be")
149
+ # with_encrypted_value(:ci_searchable_column_name, "encrypted").
150
+ # first
151
+ #
152
+ # = Encryption Key Rotation
153
+ #
154
+ # To rotate encryption keys, add a new key above the existing key, with a new key ID:
155
+ #
156
+ # Sequel::Model.plugin :column_encryption do |enc|
157
+ # enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
158
+ # enc.key 0, ENV["SEQUEL_OLD_COLUMN_ENCRYPTION_KEY"]
159
+ # end
160
+ #
161
+ # Newly encrypted data will then use the new key. Records encrypted with the older key
162
+ # will still be decrypted correctly.
163
+ #
164
+ # To force reencryption for existing records that are using the older key, you can use
165
+ # the +needing_reencryption+ dataset method and the +reencrypt+ instance method. For a
166
+ # small number of records, you can probably do:
167
+ #
168
+ # ConfidentialModel.needing_reencryption.all(&:reencrypt)
169
+ #
170
+ # With more than a small number of records, you'll want to do this in batches. It's
171
+ # possible you could use an approach such as:
172
+ #
173
+ # ds = ConfidentialModel.needing_reencryption.limit(100)
174
+ # true until ds.all(&:reencrypt).empty?
175
+ #
176
+ # After all values have been reencrypted for all models, and no models use the older
177
+ # encryption key, you can remove it from the configuration:
178
+ #
179
+ # Sequel::Model.plugin :column_encryption do |enc|
180
+ # enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
181
+ # end
182
+ #
183
+ # Once an encryption key has been removed, after no data uses it, it is safe to reuse
184
+ # the same key id for a new key. This approach allows for up to 256 concurrent keys
185
+ # in the same configuration.
186
+ #
187
+ # = Encrypting Additional Formats
188
+ #
189
+ # By default, the column_encryption plugin assumes that the decrypted data should be
190
+ # returned as a string, and a string will be passed to encrypt. However, using the
191
+ # +:format+ option, you can specify an alternate format. For example, if you want to
192
+ # encrypt a JSON representation of the object, so that you can deal with an array/hash
193
+ # and automatically have it serialized with JSON and then encrypted when saving, and
194
+ # then deserialized with JSON after decryption when it is retrieved:
195
+ #
196
+ # require 'json'
197
+ # ConfidentialModel.plugin :column_encryption do |enc|
198
+ # enc.key 0, ENV["SEQUEL_MODEL_SPECIFIC_ENCRYPTION_KEY"]
199
+ #
200
+ # enc.column :encrypted_column_name
201
+ # enc.column :searchable_column_name, searchable: true
202
+ # enc.column :ci_searchable_column_name, searchable: :case_insensitive
203
+ # enc.column :encrypted_json_column_name, format: :json
204
+ # end
205
+ #
206
+ # The values of the +:format+ are the same values you can pass as the first argument
207
+ # to +serialize_attributes+ (in the serialization plugin). You can pass an array
208
+ # with the serializer and deserializer for custom support.
209
+ #
210
+ # You can use both +:searchable+ and +:format+ together for searchable encrypted
211
+ # serialized columns. However, note that this allows only exact searches of the
212
+ # serialized version of the data. So for JSON, a search for <tt>{'a'=>1, 'b'=>2}</tt>
213
+ # would not match <tt>{'b'=>2, 'a'=>1}</tt> even though the objects are considered
214
+ # equal. If this is an issue, make sure you use a serialization format where all
215
+ # equal objects are serialized to the same string.
216
+ #
217
+ # = Enforcing Uniqueness
218
+ #
219
+ # You cannot enforce uniqueness of unencrypted data at the database level
220
+ # if you also want to support key rotation. However, absent key rotation, a
221
+ # unique index on the first 48 characters of the encrypted column can enforce uniqueness,
222
+ # as long as the column is searchable. If the encrypted column is case-insensitive
223
+ # searchable, the uniqueness is case insensitive as well.
224
+ #
225
+ # = Column Value Cryptography/Format
226
+ #
227
+ # Column values used by this plugin use the following format (+key+ is specified
228
+ # in the plugin configuration and must be exactly 32 bytes):
229
+ #
230
+ # column_value :: urlsafe_base64(flags + NUL + key_id + NUL + search_data + key_data +
231
+ # cipher_iv + cipher_auth_tag + encrypted_data)
232
+ # flags :: 1 byte, the type of record (0: not searchable, 1: searchable, 2: lowercase searchable)
233
+ # NUL :: 1 byte, ASCII NUL
234
+ # key_id :: 1 byte, the key id, supporting 256 concurrently active keys (0 - 255)
235
+ # search_data :: 0 bytes if flags is 0, 32 bytes if flags is 1 or 2.
236
+ # Format is HMAC-SHA256(key, unencrypted_data).
237
+ # Ignored on decryption, only used for searching.
238
+ # key_data :: 32 bytes random data used to construct cipher key
239
+ # cipher_iv :: 12 bytes, AES-256-GCM cipher random initialization vector
240
+ # cipher_auth_tag :: 16 bytes, AES-256-GCM cipher authentication tag
241
+ # encrypted_data :: AES-256-GCM(HMAC-SHA256(key, key_data),
242
+ # padding_size + padding + unencrypted_data)
243
+ # padding_size :: 1 byte, with the amount of padding (0-255 bytes of padding allowed)
244
+ # padding :: number of bytes specified by padding size, ignored on decryption
245
+ # unencrypted_data :: actual column value
246
+ #
247
+ # The reason for <tt>flags + NUL + key_id + NUL</tt> (4 bytes) as the header is to allow for
248
+ # an easy way to search for values needing reencryption using a database index. It takes
249
+ # the first three bytes and converts them to base64, and looks for values less than that value
250
+ # or greater than that value with 'B' appended. The NUL byte in the fourth byte of the header
251
+ # ensures that after base64 encoding, the fifth byte in the column will be 'A'.
252
+ #
253
+ # The reason for <tt>search_data</tt> (32 bytes) directly after is that for searchable values,
254
+ # after base64 encoding of the header and search data, it is 48 bytes and can be used directly
255
+ # as a prefix search on the column, which can be supported by the same database index. This is
256
+ # more efficient than a full column value search for large values, and allows for case-insensitive
257
+ # searching without a separate column, by having the search_data be based on the lowercase value
258
+ # while the unencrypted data is original case.
259
+ #
260
+ # The reason for the padding is so that a database administrator cannot be sure exactly how
261
+ # many bytes are in the column. It is stored encrypted because otherwise the database
262
+ # administrator could calculate it by decoding the base64 data.
263
+ #
264
+ # = Unsupported Features
265
+ #
266
+ # The following features are delibrately not supported:
267
+ #
268
+ # == Compression
269
+ #
270
+ # Allowing compression with encryption is inviting security issues later.
271
+ # While padding can reduce the risk of compression with encryption, it does not
272
+ # eliminate it entirely. Users that must have compression with encryption can use
273
+ # the +:format+ option with a serializer that compresses and a deserializer that
274
+ # decompresses.
275
+ #
276
+ # == Mixing Encrypted/Unencrypted Data
277
+ #
278
+ # Mixing encrypted and unencrypted data increases the complexity and security risk, since there
279
+ # is a chance unencrypted data could look like encrypted data in the pathologic case.
280
+ # If you have existing unencrypted data that would like to encrypt, create a new column for
281
+ # the encrypted data, and then migrate the data from the unencrypted column to the encrypted
282
+ # column. After all unencrypted values have been migrated, drop the unencrypted column.
283
+ #
284
+ # == Arbitrary Encryption Schemes
285
+ #
286
+ # Supporting arbitrary encryption schemes increases the complexity risk.
287
+ # If in the future AES-256-GCM is not considered a secure enough cipher, it is possible to
288
+ # extend the current format using the reserved values in the first two bytes of the header.
289
+ #
290
+ # = Caveats
291
+ #
292
+ # As column_encryption is a model plugin, it only works with using model instance methods.
293
+ # If you directly modify the database using a dataset or an external program that modifies
294
+ # the contents of the encrypted columns, you will probably corrupt the data. To make data
295
+ # corruption less likely, it is best to have a CHECK constraints on the encrypted column
296
+ # with a basic format and length check:
297
+ #
298
+ # DB.alter_table(:table_name) do
299
+ # c = Sequel[:encrypted_column_name]
300
+ # add_constraint(:encrypted_column_name_format,
301
+ # c.like('AA__A%') | c.like('Ag__A%') | c.like('AQ__A%'))
302
+ # add_constraint(:encrypted_column_name_length, Sequel.char_length(c) >= 88)
303
+ # end
304
+ #
305
+ # If possible, it's also best to check that the column is valid urlsafe base64 data of
306
+ # sufficient length. This can be done on PostgreSQL using a combination of octet_length,
307
+ # decode, and regexp_replace:
308
+ #
309
+ # DB.alter_table(:ce_test) do
310
+ # c = Sequel[:encrypted_column_name]
311
+ # add_constraint(:enc_base64) do
312
+ # octet_length(decode(regexp_replace(regexp_replace(c, '_', '/', 'g'), '-', '+', 'g'), 'base64')) >= 65}
313
+ # end
314
+ # end
315
+ #
316
+ # Such constraints will probably be sufficient to protect against most unintentional corruption of
317
+ # encrypted columns.
318
+ #
319
+ # If the database supports transparent data encryption and you trust the database administrator,
320
+ # using the database support is probably a better approach.
321
+ #
322
+ # The column_encryption plugin is only supported on Ruby 2.3+ and when the Ruby openssl standard
323
+ # library supports the AES-256-GCM cipher.
324
+ module ColumnEncryption
325
+ # Cryptor handles the encryption and decryption of rows for a key set.
326
+ # It also provides methods that return search prefixes, which datasets
327
+ # use in queries.
328
+ #
329
+ # The same cryptor can support non-searchable, searchable, and case-insensitive
330
+ # searchable columns.
331
+ class Cryptor # :nodoc:
332
+ # Flags
333
+ NOT_SEARCHABLE = 0
334
+ SEARCHABLE = 1
335
+ LOWERCASE_SEARCHABLE = 2
336
+
337
+ # This is the default padding, but up to 2x the padding can be used for a record.
338
+ DEFAULT_PADDING = 8
339
+
340
+ # Keys should be an array of arrays containing key_id, key string, auth_data, and padding.
341
+ def initialize(keys)
342
+ if keys.empty?
343
+ raise Error, "Cannot initialize encryptor without encryption key"
344
+ end
345
+
346
+ # First key is used for encryption
347
+ @key_id, @key, @auth_data, @padding = keys[0]
348
+
349
+ # All keys are candidates for decryption
350
+ @key_map = {}
351
+ keys.each do |key_id, key, auth_data, padding|
352
+ @key_map[key_id] = [key, auth_data, padding].freeze
353
+ end
354
+
355
+ freeze
356
+ end
357
+
358
+ # Decrypt using any supported format and any available key.
359
+ def decrypt(data)
360
+ begin
361
+ data = Base64.urlsafe_decode64(data)
362
+ rescue ArgumentError
363
+ raise Error, "Unable to decode encrypted column: invalid base64"
364
+ end
365
+
366
+ unless data.getbyte(1) == 0 && data.getbyte(3) == 0
367
+ raise Error, "Unable to decode encrypted column: invalid format"
368
+ end
369
+
370
+ flags = data.getbyte(0)
371
+
372
+ key, auth_data = @key_map[data.getbyte(2)]
373
+ unless key
374
+ raise Error, "Unable to decode encrypted column: invalid key id"
375
+ end
376
+
377
+ case flags
378
+ when NOT_SEARCHABLE
379
+ if data.bytesize < 65
380
+ raise Error, "Decoded encrypted column smaller than minimum size"
381
+ end
382
+
383
+ data.slice!(0, 4)
384
+ when SEARCHABLE, LOWERCASE_SEARCHABLE
385
+ if data.bytesize < 97
386
+ raise Error, "Decoded encrypted column smaller than minimum size"
387
+ end
388
+
389
+ data.slice!(0, 36)
390
+ else
391
+ raise Error, "Unable to decode encrypted column: invalid flags"
392
+ end
393
+
394
+ key_part = data.slice!(0, 32)
395
+ cipher_iv = data.slice!(0, 12)
396
+ auth_tag = data.slice!(0, 16)
397
+
398
+ cipher = OpenSSL::Cipher.new("aes-256-gcm")
399
+ cipher.decrypt
400
+ cipher.iv = cipher_iv
401
+ cipher.key = OpenSSL::HMAC.digest(OpenSSL::Digest::SHA256.new, key, key_part)
402
+ cipher.auth_data = auth_data
403
+ cipher.auth_tag = auth_tag
404
+ begin
405
+ decrypted_data = cipher.update(data) << cipher.final
406
+ rescue OpenSSL::Cipher::CipherError => e
407
+ raise Error, "Unable to decrypt encrypted column: #{e.class} (probably due to encryption key or auth data mismatch or corrupt data)"
408
+ end
409
+
410
+ # Remove padding
411
+ decrypted_data.slice!(0, decrypted_data.getbyte(0) + 1)
412
+
413
+ decrypted_data
414
+ end
415
+
416
+ # Encrypt in not searchable format with the first configured encryption key.
417
+ def encrypt(data)
418
+ _encrypt(data, "#{NOT_SEARCHABLE.chr}\0#{@key_id.chr}\0")
419
+ end
420
+
421
+ # Encrypt in searchable format with the first configured encryption key.
422
+ def searchable_encrypt(data)
423
+ _encrypt(data, _search_prefix(data, SEARCHABLE, @key_id, @key))
424
+ end
425
+
426
+ # Encrypt in case insensitive searchable format with the first configured encryption key.
427
+ def case_insensitive_searchable_encrypt(data)
428
+ _encrypt(data, _search_prefix(data.downcase, LOWERCASE_SEARCHABLE, @key_id, @key))
429
+ end
430
+
431
+ # The prefix string of columns for the given search type and the first configured encryption key.
432
+ # Used to find values that do not use this prefix in order to perform reencryption.
433
+ def current_key_prefix(search_type)
434
+ Base64.urlsafe_encode64("#{search_type.chr}\0#{@key_id.chr}")
435
+ end
436
+
437
+ # The prefix values to search for the given data (an array of strings), assuming the column uses
438
+ # the searchable format.
439
+ def search_prefixes(data)
440
+ _search_prefixes(data, SEARCHABLE)
441
+ end
442
+
443
+ # The prefix values to search for the given data (an array of strings), assuming the column uses
444
+ # the case insensitive searchable format.
445
+ def lowercase_search_prefixes(data)
446
+ _search_prefixes(data.downcase, LOWERCASE_SEARCHABLE)
447
+ end
448
+
449
+ # The prefix values to search for the given data (an array of strings), assuming the column uses
450
+ # either the searchable or the case insensitive searchable format. Should be used only when
451
+ # transitioning between formats (used by the :search_both option when encrypting columns).
452
+ def regular_and_lowercase_search_prefixes(data)
453
+ search_prefixes(data) + lowercase_search_prefixes(data)
454
+ end
455
+
456
+ private
457
+
458
+ # An array of strings, one for each configured encryption key, to find encypted values matching
459
+ # the given data and search format.
460
+ def _search_prefixes(data, search_type)
461
+ @key_map.map do |key_id, (key, _)|
462
+ Base64.urlsafe_encode64(_search_prefix(data, search_type, key_id, key))
463
+ end
464
+ end
465
+
466
+ # The prefix to use for searchable data, including the HMAC-SHA256(key, data).
467
+ def _search_prefix(data, search_type, key_id, key)
468
+ "#{search_type.chr}\0#{key_id.chr}\0#{OpenSSL::HMAC.digest(OpenSSL::Digest::SHA256.new, key, data)}"
469
+ end
470
+
471
+ # Encrypt the data using AES-256-GCM, with the given prefix.
472
+ def _encrypt(data, prefix)
473
+ padding = @padding
474
+ random_data = SecureRandom.random_bytes(32)
475
+ cipher = OpenSSL::Cipher.new("aes-256-gcm")
476
+ cipher.encrypt
477
+ cipher.key = OpenSSL::HMAC.digest(OpenSSL::Digest::SHA256.new, @key, random_data)
478
+ cipher_iv = cipher.random_iv
479
+ cipher.auth_data = @auth_data
480
+
481
+ cipher_text = String.new
482
+ data_size = data.bytesize
483
+
484
+ padding_size = if padding
485
+ (padding * rand(1)) + padding - (data.bytesize % padding)
486
+ else
487
+ 0
488
+ end
489
+
490
+ cipher_text << cipher.update(padding_size.chr)
491
+ cipher_text << cipher.update(SecureRandom.random_bytes(padding_size)) if padding_size > 0
492
+ cipher_text << cipher.update(data) if data_size > 0
493
+ cipher_text << cipher.final
494
+
495
+ Base64.urlsafe_encode64("#{prefix}#{random_data}#{cipher_iv}#{cipher.auth_tag}#{cipher_text}")
496
+ end
497
+ end
498
+
499
+ # The object type yielded to blocks passed to the +column+ method inside
500
+ # <tt>plugin :column_encryption</tt> blocks. This is used to configure custom
501
+ # per-column keys.
502
+ class ColumnDSL # :nodoc:
503
+ # An array of arrays for the data for the keys configured inside the block.
504
+ attr_reader :keys
505
+
506
+ def initialize
507
+ @keys = []
508
+ end
509
+
510
+ # Verify that the key_id, key, and options are value.
511
+ def key(key_id, key, opts=OPTS)
512
+ unless key_id.is_a?(Integer) && key_id >= 0 && key_id <= 255
513
+ raise Error, "invalid key_id argument, must be integer between 0 and 255"
514
+ end
515
+
516
+ unless key.is_a?(String) && key.bytesize == 32
517
+ raise Error, "invalid key argument, must be string with exactly 32 bytes"
518
+ end
519
+
520
+ if opts.has_key?(:padding)
521
+ if padding = opts[:padding]
522
+ unless padding.is_a?(Integer) && padding >= 1 && padding <= 120
523
+ raise Error, "invalid :padding option, must be between 1 and 120"
524
+ end
525
+ end
526
+ else
527
+ padding = Cryptor::DEFAULT_PADDING
528
+ end
529
+
530
+ @keys << [key_id, key, opts[:auth_data].to_s, padding].freeze
531
+ end
532
+ end
533
+
534
+ # The object type yielded to <tt>plugin :column_encryption</tt> blocks,
535
+ # used to configure encryption keys and encrypted columns.
536
+ class DSL < ColumnDSL # :nodoc:
537
+ # An array of arrays of data for the columns configured inside the block.
538
+ attr_reader :columns
539
+
540
+ def initialize
541
+ super
542
+ @columns = []
543
+ end
544
+
545
+ # Store the column information.
546
+ def column(column, opts=OPTS, &block)
547
+ @columns << [column, opts, block].freeze
548
+ end
549
+ end
550
+
551
+ def self.apply(model, opts=OPTS)
552
+ model.plugin :serialization
553
+ end
554
+
555
+ def self.configure(model)
556
+ dsl = DSL.new
557
+ yield dsl
558
+
559
+ model.instance_exec do
560
+ unless dsl.keys.empty?
561
+ @column_encryption_keys = dsl.keys.freeze
562
+ @column_encryption_cryptor = nil
563
+ end
564
+
565
+ @column_encryption_metadata = Hash[@column_encryption_metadata || {}]
566
+
567
+ dsl.columns.each do |column, opts, block|
568
+ _encrypt_column(column, opts, &block)
569
+ end
570
+
571
+ @column_encryption_metadata.freeze
572
+ end
573
+ end
574
+
575
+ # This stores four callables for handling encyption, decryption, data searching,
576
+ # and key searching. One of these is created for each encrypted column.
577
+ ColumnEncryptionMetadata = Struct.new(:encryptor, :decryptor, :data_searcher, :key_searcher) # :nodoc:
578
+
579
+ module ClassMethods
580
+ private
581
+
582
+ # A hash with column symbol keys and ColumnEncryptionMetadata values for each
583
+ # encrypted column.
584
+ attr_reader :column_encryption_metadata
585
+
586
+ # The default Cryptor to use for encrypted columns. This is only overridden if
587
+ # per-column keys are used.
588
+ def column_encryption_cryptor
589
+ @column_encryption_cryptor ||= Cryptor.new(@column_encryption_keys)
590
+ end
591
+
592
+ # Setup encryption for the given column.
593
+ def _encrypt_column(column, opts)
594
+ cryptor ||= if block_given?
595
+ dsl = ColumnDSL.new
596
+ yield dsl
597
+ Cryptor.new(dsl.keys)
598
+ else
599
+ column_encryption_cryptor
600
+ end
601
+
602
+ encrypt_method, search_prefixes_method, search_type = case searchable = opts[:searchable]
603
+ when nil, false
604
+ [:encrypt, nil, Cryptor::NOT_SEARCHABLE]
605
+ when true
606
+ [:searchable_encrypt, :search_prefixes, Cryptor::SEARCHABLE]
607
+ when :case_insensitive
608
+ [:case_insensitive_searchable_encrypt, :lowercase_search_prefixes, Cryptor::LOWERCASE_SEARCHABLE]
609
+ else
610
+ raise Error, "invalid :searchable option for encrypted column: #{searchable.inspect}"
611
+ end
612
+
613
+ if searchable && opts[:search_both]
614
+ search_prefixes_method = :regular_and_lowercase_search_prefixes
615
+ end
616
+
617
+ # Setup the callables used in the metadata.
618
+ encryptor = cryptor.method(encrypt_method)
619
+ decryptor = cryptor.method(:decrypt)
620
+ data_searcher = cryptor.method(search_prefixes_method) if search_prefixes_method
621
+ key_searcher = lambda{cryptor.current_key_prefix(search_type)}
622
+
623
+ if format = opts[:format]
624
+ if format.is_a?(Symbol)
625
+ unless format = Sequel.synchronize{Serialization::REGISTERED_FORMATS[format]}
626
+ raise(Error, "Unsupported serialization format: #{format} (valid formats: #{Sequel.synchronize{Serialization::REGISTERED_FORMATS.keys}.inspect})")
627
+ end
628
+ end
629
+
630
+ # If a custom serialization format is used, override the
631
+ # callables to handle serialization and deserialization.
632
+ serializer, deserializer = format
633
+ enc, dec, data_s = encryptor, decryptor, data_searcher
634
+ encryptor = lambda do |data|
635
+ enc.call(serializer.call(data))
636
+ end
637
+ decryptor = lambda do |data|
638
+ deserializer.call(dec.call(data))
639
+ end
640
+ data_searcher = lambda do |data|
641
+ data_s.call(serializer.call(data))
642
+ end
643
+ end
644
+
645
+ # Setup the setter and getter methods to do encryption and decryption using
646
+ # the serialization plugin.
647
+ serialize_attributes([encryptor, decryptor], column)
648
+
649
+ column_encryption_metadata[column] = ColumnEncryptionMetadata.new(encryptor, decryptor, data_searcher, key_searcher).freeze
650
+
651
+ nil
652
+ end
653
+ end
654
+
655
+ module ClassMethods
656
+ Plugins.def_dataset_methods(self, [:with_encrypted_value, :needing_reencryption])
657
+
658
+ Plugins.inherited_instance_variables(self,
659
+ :@column_encryption_cryptor=>nil,
660
+ :@column_encryption_keys=>nil,
661
+ :@column_encryption_metadata=>nil,
662
+ )
663
+ end
664
+
665
+ module InstanceMethods
666
+ # Reencrypt the model if needed. Looks at all of the models encrypted columns
667
+ # and if any were encypted with older keys or a different format, reencrypt
668
+ # with the current key and format and save the object. Returns the object
669
+ # if reencryption was needed, or nil if reencryption was not needed.
670
+ def reencrypt
671
+ do_save = false
672
+
673
+ model.send(:column_encryption_metadata).each do |column, metadata|
674
+ if (value = values[column]) && !value.start_with?(metadata.key_searcher.call)
675
+ do_save = true
676
+ values[column] = metadata.encryptor.call(metadata.decryptor.call(value))
677
+ end
678
+ end
679
+
680
+ save if do_save
681
+ end
682
+ end
683
+
684
+ module DatasetMethods
685
+ # Filter the dataset to only match rows where the column contains an encrypted version
686
+ # of value. Only works on searchable encrypted columns.
687
+ def with_encrypted_value(column, value)
688
+ metadata = model.send(:column_encryption_metadata)[column]
689
+
690
+ unless metadata && metadata.data_searcher
691
+ raise Error, "lookup for encrypted column #{column.inspect} is not supported"
692
+ end
693
+
694
+ prefixes = metadata.data_searcher.call(value)
695
+ where(Sequel.|(*prefixes.map{|v| Sequel.like(column, "#{escape_like(v)}%")}))
696
+ end
697
+
698
+ # Filter the dataset to exclude rows where all encrypted columns are already encrypted
699
+ # with the current key and format.
700
+ def needing_reencryption
701
+ incorrect_column_prefixes = model.send(:column_encryption_metadata).map do |column, metadata|
702
+ prefix = metadata.key_searcher.call
703
+ (Sequel[column] < prefix) | (Sequel[column] > prefix + 'B')
704
+ end
705
+
706
+ where(Sequel.|(*incorrect_column_prefixes))
707
+ end
708
+ end
709
+ end
710
+ end
711
+ end
@@ -127,7 +127,7 @@ module Sequel
127
127
  def serialize_attributes(format, *columns)
128
128
  if format.is_a?(Symbol)
129
129
  unless format = Sequel.synchronize{REGISTERED_FORMATS[format]}
130
- raise(Error, "Unsupported serialization format: #{format} (valid formats: #{Sequel.synchronize{REGISTERED_FORMATS.keys}.map(&:inspect).join})")
130
+ raise(Error, "Unsupported serialization format: #{format} (valid formats: #{Sequel.synchronize{REGISTERED_FORMATS.keys}.inspect})")
131
131
  end
132
132
  end
133
133
  serializer, deserializer = format
@@ -154,7 +154,10 @@ module Sequel
154
154
  deserialized_values[column] = deserialize_value(column, super())
155
155
  end
156
156
  end
157
- define_method("#{column}=") do |v|
157
+ alias_method(column, column)
158
+
159
+ setter = :"#{column}="
160
+ define_method(setter) do |v|
158
161
  cc = changed_columns
159
162
  if !cc.include?(column) && (new? || get_column_value(column) != v)
160
163
  cc << column
@@ -164,6 +167,7 @@ module Sequel
164
167
 
165
168
  deserialized_values[column] = v
166
169
  end
170
+ alias_method(setter, setter)
167
171
  end
168
172
  end
169
173
  end
@@ -6,7 +6,7 @@ module Sequel
6
6
 
7
7
  # The minor version of Sequel. Bumped for every non-patch level
8
8
  # release, generally around once a month.
9
- MINOR = 42
9
+ MINOR = 43
10
10
 
11
11
  # The tiny version of Sequel. Usually 0, only bumped for bugfix
12
12
  # releases that fix regressions from previous versions.
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sequel
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.42.0
4
+ version: 5.43.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jeremy Evans
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-03-01 00:00:00.000000000 Z
11
+ date: 2021-04-01 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: minitest
@@ -186,6 +186,7 @@ extra_rdoc_files:
186
186
  - doc/release_notes/5.40.0.txt
187
187
  - doc/release_notes/5.41.0.txt
188
188
  - doc/release_notes/5.42.0.txt
189
+ - doc/release_notes/5.43.0.txt
189
190
  - doc/release_notes/5.5.0.txt
190
191
  - doc/release_notes/5.6.0.txt
191
192
  - doc/release_notes/5.7.0.txt
@@ -256,6 +257,7 @@ files:
256
257
  - doc/release_notes/5.40.0.txt
257
258
  - doc/release_notes/5.41.0.txt
258
259
  - doc/release_notes/5.42.0.txt
260
+ - doc/release_notes/5.43.0.txt
259
261
  - doc/release_notes/5.5.0.txt
260
262
  - doc/release_notes/5.6.0.txt
261
263
  - doc/release_notes/5.7.0.txt
@@ -459,6 +461,7 @@ files:
459
461
  - lib/sequel/plugins/caching.rb
460
462
  - lib/sequel/plugins/class_table_inheritance.rb
461
463
  - lib/sequel/plugins/column_conflicts.rb
464
+ - lib/sequel/plugins/column_encryption.rb
462
465
  - lib/sequel/plugins/column_select.rb
463
466
  - lib/sequel/plugins/columns_updated.rb
464
467
  - lib/sequel/plugins/composition.rb