lockbox 0.4.2 → 0.4.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cb4f05b7519d563c465c963893ded5f21063f666bdb41d4bfc69f23ea6b82374
-  data.tar.gz: 0f13b613b16b15ca6514de06fc57ec62d98b94a06a01eb6d8e77fdce9d23343b
+  metadata.gz: 3f8c447dd90537203a1a3038c347c10de0e48f5b29795b382a2a77019e6e5764
+  data.tar.gz: 9133e9eb0c2132b7c77c39f8c24a3c27ea9b3cbb1d3d82f7f069b2db9992198f
 SHA512:
-  metadata.gz: bdb2f4cbf5977bd1e40c9055c6e166c819b8e13ccbdd2d870c80ee907608205f72f65faa9fe56e882fcf9eab155b506537f29bd7593c3fcf31e3a0f94740e0ec
-  data.tar.gz: ef966eff17c633ee5b590742c33a107bd58da4d262fbaf1bea458c2e5146dc16993f75b6d783118f3ff7f7fd827913423c71558bf1e8d60df66e3de35627e1a1
+  metadata.gz: 4396ee4ead0de0592e7a3574b563f98d58f0402198dd8662cbdc374cc5f8a39e629f6397414b498456c665e8a635518db3a532056ecd42154206fde4ab938e5c
+  data.tar.gz: 6057ea6f43db261580a0ee0ae13f1303251d2ee8ef2b1bcdba8399b6a858dca1fdf6c84e25f5da85a059a6d260be7094969e2118ff11a8c1d2565617c48f74cd

data/CHANGELOG.md

@@ -1,3 +1,27 @@
+## 0.4.7 (2020-08-18)
+
+- Added `lockbox_options` method to encrypted CarrierWave uploaders
+- Improved attribute loading when no decryption key specified
+
+## 0.4.6 (2020-07-02)
+
+- Added support for `update_column` and `update_columns`
+
+## 0.4.5 (2020-06-26)
+
+- Improved error message for non-string values
+- Fixed error with migrating Action Text
+- Fixed error with migrating serialized attributes
+
+## 0.4.4 (2020-06-23)
+
+- Added support for `pluck`
+
+## 0.4.3 (2020-05-26)
+
+- Improved error message for bad key length
+- Fixed missing attribute error
+
 ## 0.4.2 (2020-05-11)
 
 - Added experimental support for migrating Active Storage files

data/README.md

@@ -190,6 +190,8 @@ end
 
 ## Action Text
 
+**Note:** Action Text uses direct uploads for files, which cannot be encrypted with application-level encryption like Lockbox. This only encrypts the database field.
+
 Create a migration with:
 
 ```ruby
@@ -264,8 +266,9 @@ end
 
 There are a few limitations to be aware of:
 
-- Metadata like image width and height are not extracted when encrypted
-- Direct uploads cannot be encrypted
+- Variants and previews aren’t supported when encrypted
+- Metadata like image width and height aren’t extracted when encrypted
+- Direct uploads can’t be encrypted with application-level encryption like Lockbox, but can use server-side encryption
 
 To serve encrypted files, use a controller action.
 
@@ -278,7 +281,7 @@ end
 
 #### Migrating Existing Files [experimental]
 
-**Note:** This feature is experimental. Please try it in a non-production environment and let us know how it goes.
+**Note:** This feature is experimental. Please try it in a non-production environment and [share](https://github.com/ankane/lockbox/issues/44) how it goes.
 
 Lockbox makes it easy to encrypt existing files without downtime.
 
@@ -508,6 +511,24 @@ Lockbox.rotate(User, attributes: [:email])
 
 Once all records are rotated, you can remove `previous_versions` from the model.
 
+### Action Text
+
+Update your initializer:
+
+```ruby
+Lockbox.encrypts_action_text_body(previous_versions: [{key: previous_key}])
+```
+
+Use `master_key` instead of `key` if passing the master key.
+
+To rotate existing records, use:
+
+```ruby
+Lockbox.rotate(ActionText::RichText, attributes: [:body])
+```
+
+Once all records are rotated, you can remove `previous_versions` from the initializer.
+
 ### Active Storage
 
 Update your model:
@@ -550,6 +571,14 @@ User.find_each do |user|
 end
 ```
 
+For multiple files, use:
+
+```ruby
+User.find_each do |user|
+  user.licenses.map(&:rotate_encryption!)
+end
+```
+
 Once all files are rotated, you can remove `previous_versions` from the model.
 
 ### Local Files & Strings
@@ -734,12 +763,32 @@ end
 
 You can use a key management service to manage your keys with [KMS Encrypted](https://github.com/ankane/kms_encrypted).
 
+For Active Record and Mongoid, use:
+
 ```ruby
 class User < ApplicationRecord
   encrypts :email, key: :kms_key
 end
 ```
 
+For Action Text, use:
+
+```ruby
+ActiveSupport.on_load(:action_text_rich_text) do
+  ActionText::RichText.has_kms_key
+end
+
+Lockbox.encrypts_action_text_body(key: :kms_key)
+```
+
+For Active Storage, use:
+
+```ruby
+class User < ApplicationRecord
+  encrypts_attached :license, key: :kms_key
+end
+```
+
 For CarrierWave, use:
 
 ```ruby
@@ -772,7 +821,7 @@ lockbox.encrypt("clear").bytesize     # 44
 lockbox.encrypt("consider").bytesize  # 44
 ```
 
-The block size for padding is 16 bytes by default. If we have a status larger than 15 bytes, it will have a different length than the others.
+The block size for padding is 16 bytes by default. Lockbox uses [ISO/IEC 7816-4](https://en.wikipedia.org/wiki/Padding_(cryptography)#ISO/IEC_7816-4) padding, which uses at least one byte, so if we have a status larger than 15 bytes, it will have a different length than the others.
 
 ```ruby
 box.encrypt("length15status!").bytesize   # 44
@@ -785,9 +834,25 @@ Change the block size with:
 Lockbox.new(padding: 32) # bytes
 ```
 
+## Associated Data
+
+You can pass extra context during encryption to make sure encrypted data isn’t moved to a different context.
+
+```ruby
+lockbox = Lockbox.new(key: key)
+ciphertext = lockbox.encrypt(message, associated_data: "somecontext")
+```
+
+Without the same context, decryption will fail.
+
+```ruby
+lockbox.decrypt(ciphertext, associated_data: "somecontext")  # success
+lockbox.decrypt(ciphertext, associated_data: "othercontext") # fails
+```
+
 ## Binary Columns
 
-You can use `binary` columns for the ciphertext instead of `text` columns to save space.
+You can use `binary` columns for the ciphertext instead of `text` columns.
 
 ```ruby
 class AddEmailCiphertextToUsers < ActiveRecord::Migration[6.0]
@@ -797,7 +862,7 @@ class AddEmailCiphertextToUsers < ActiveRecord::Migration[6.0]
 end
 ```
 
-You should disable Base64 encoding if you do this.
+Disable Base64 encoding to save space.
 
 ```ruby
 class User < ApplicationRecord
@@ -965,3 +1030,5 @@ cd lockbox
 bundle install
 bundle exec rake test
 ```
+
+For security issues, send an email to the address on [this page](https://github.com/ankane).

data/SECURITY.md

@@ -0,0 +1,3 @@
+# Security Policy
+
+For security issues, send an email to the address on [this page](https://github.com/ankane).

data/lib/lockbox.rb

@@ -5,6 +5,7 @@ require "securerandom"
 
 # modules
 require "lockbox/box"
+require "lockbox/calculations"
 require "lockbox/encryptor"
 require "lockbox/key_generator"
 require "lockbox/io"
@@ -25,6 +26,7 @@ if defined?(ActiveSupport)
   ActiveSupport.on_load(:active_record) do
     extend Lockbox::Model
     extend Lockbox::Model::Attached
+    ActiveRecord::Calculations.prepend Lockbox::Calculations
   end
 
   ActiveSupport.on_load(:mongoid) do

data/lib/lockbox/active_storage_extensions.rb

@@ -1,7 +1,22 @@
-# ideally encrypt and decrypt would happen at the blob/service level
-# however, there isn't really a great place to define encryption settings there
-# instead, we encrypt and decrypt at the attachment level,
-# and we define encryption settings at the model level
+# Ideally encryption and decryption would happen at the blob/service level.
+# However, Active Storage < 6.1 only supports a single service (per environment).
+# This means all attachments need to be encrypted or none of them,
+# which is often not practical.
+#
+# Active Storage 6.1 adds support for multiple services, which changes this.
+# We could have a Lockbox service:
+#
+# lockbox:
+#   service: Lockbox
+#   backend: local    # delegate to another service, like mirror service
+#   key:     ...      # Lockbox options
+#
+# However, the checksum is computed *and stored on the blob*
+# before the file is passed to the service.
+# We don't want the MD5 checksum of the plaintext stored in the database.
+#
+# Instead, we encrypt and decrypt at the attachment level,
+# and we define encryption settings at the model level.
 module Lockbox
   module ActiveStorageExtensions
     module Attached
@@ -95,6 +110,16 @@ module Lockbox
         result
       end
 
+      def variant(*args)
+        raise Lockbox::Error, "Variant not supported for encrypted files" if Utils.encrypted_options(record, name)
+        super
+      end
+
+      def preview(*args)
+        raise Lockbox::Error, "Preview not supported for encrypted files" if Utils.encrypted_options(record, name)
+        super
+      end
+
       if ActiveStorage::VERSION::MAJOR >= 6
         def open(**options)
           blob.open(**options) do |file|

data/lib/lockbox/calculations.rb

@@ -0,0 +1,36 @@
+module Lockbox
+  module Calculations
+    def pluck(*column_names)
+      return super unless model.respond_to?(:lockbox_attributes)
+
+      lockbox_columns = column_names.map.with_index { |c, i| [model.lockbox_attributes[c.to_sym], i] }.select(&:first)
+      return super unless lockbox_columns.any?
+
+      # replace column with ciphertext column
+      lockbox_columns.each do |la, i|
+        column_names[i] = la[:encrypted_attribute]
+      end
+
+      # pluck
+      result = super(*column_names)
+
+      # decrypt result
+      # handle pluck to single columns and multiple
+      #
+      # we can't pass context to decrypt method
+      # so this won't work if any options are a symbol or proc
+      if column_names.size == 1
+        la = lockbox_columns.first.first
+        result.map! { |v| model.send("decrypt_#{la[:encrypted_attribute]}", v) }
+      else
+        lockbox_columns.each do |la, i|
+          result.each do |v|
+            v[i] = model.send("decrypt_#{la[:encrypted_attribute]}", v[i])
+          end
+        end
+      end
+
+      result
+    end
+  end
+end

data/lib/lockbox/carrier_wave_extensions.rb

@@ -2,18 +2,32 @@ module Lockbox
   module CarrierWaveExtensions
     def encrypt(**options)
       class_eval do
+        # uses same hook as process (before cache)
+        # processing can be disabled, so better to keep separate
         before :cache, :encrypt
 
+        define_singleton_method :lockbox_options do
+          options
+        end
+
         def encrypt(file)
-          @file = CarrierWave::SanitizedFile.new(with_notification("encrypt_file") { lockbox.encrypt_io(file) })
+          # safety check
+          # see CarrierWave::Uploader::Cache#cache!
+          raise Lockbox::Error, "Expected files to be equal. Please report an issue." unless file && @file && file == @file
+
+          # processors in CarrierWave move updated file to current_path
+          # however, this causes versions to use the processed file
+          # we only want to change the file for the current version
+          @file = CarrierWave::SanitizedFile.new(lockbox_notify("encrypt_file") { lockbox.encrypt_io(file) })
         end
 
         # TODO safe to memoize?
         def read
           r = super
-          with_notification("decrypt_file") { lockbox.decrypt(r) } if r
+          lockbox_notify("decrypt_file") { lockbox.decrypt(r) } if r
         end
 
+        # use size of plaintext since read and content type use plaintext
         def size
           read.bytesize
         end
@@ -23,6 +37,7 @@ module Lockbox
           MimeMagic.by_magic(read).try(:type) || "invalid/invalid"
         end
 
+        # disable processing since already processed
         def rotate_encryption!
           io = Lockbox::IO.new(read)
           io.original_filename = file.filename
@@ -46,6 +61,8 @@ module Lockbox
           end
         end
 
+        # for mounted uploaders, use mounted name
+        # for others, use uploader name
         def lockbox_name
           if mounted_as
             mounted_as.to_s
@@ -58,7 +75,9 @@ module Lockbox
           end
         end
 
-        def with_notification(type)
+        # Active Support notifications so it's easier
+        # to see when files are encrypted and decrypted
+        def lockbox_notify(type)
           if defined?(ActiveSupport::Notifications)
             name = lockbox_name
 

data/lib/lockbox/encryptor.rb

@@ -13,7 +13,7 @@ module Lockbox
     end
 
     def encrypt(message, **options)
-      message = check_string(message, "message")
+      message = check_string(message)
       ciphertext = @boxes.first.encrypt(message, **options)
       ciphertext = Base64.strict_encode64(ciphertext) if @encode
       ciphertext
@@ -21,7 +21,7 @@ module Lockbox
 
     def decrypt(ciphertext, **options)
       ciphertext = Base64.decode64(ciphertext) if @encode
-      ciphertext = check_string(ciphertext, "ciphertext")
+      ciphertext = check_string(ciphertext)
 
       # ensure binary
       if ciphertext.encoding != Encoding::BINARY
@@ -66,9 +66,10 @@ module Lockbox
 
     private
 
-    def check_string(str, name)
+    def check_string(str)
       str = str.read if str.respond_to?(:read)
-      raise TypeError, "can't convert #{name} to string" unless str.respond_to?(:to_str)
+      # Ruby uses "no implicit conversion of Object into String"
+      raise TypeError, "can't convert #{str.class.name} to String" unless str.respond_to?(:to_str)
       str.to_str
     end
 

data/lib/lockbox/key_generator.rb

@@ -11,7 +11,7 @@ module Lockbox
       raise ArgumentError, "Missing attribute for key generation" if attribute.to_s.empty?
 
       c = "\xB4"*32
-      hkdf(Lockbox::Utils.decode_key(@master_key), salt: table.to_s, info: "#{c}#{attribute}", length: 32, hash: "sha384")
+      hkdf(Lockbox::Utils.decode_key(@master_key, name: "Master key"), salt: table.to_s, info: "#{c}#{attribute}", length: 32, hash: "sha384")
     end
 
     private

data/lib/lockbox/migrator.rb

@@ -116,6 +116,13 @@ module Lockbox
       end
     end
 
+    # there's a small chance for this process to read data,
+    # another process to update the data, and
+    # this process to write the now stale data
+    # this time window can be reduced with smaller batch sizes
+    # locking individual records could eliminate this
+    # one option is: relation.in_batches { |batch| batch.lock }
+    # which runs SELECT ... FOR UPDATE in Postgres
     def migrate_records(records, fields:, blind_indexes:, restart:, rotate:)
       # do computation outside of transaction
       # especially expensive blind index computation

data/lib/lockbox/model.rb

@@ -87,7 +87,13 @@ module Lockbox
                 # essentially a no-op if already loaded
                 # an exception is thrown if decryption fails
                 self.class.lockbox_attributes.each do |_, lockbox_attribute|
-                  send(lockbox_attribute[:attribute])
+                  # don't try to decrypt if no decryption key given
+                  next if lockbox_attribute[:algorithm] == "hybrid" && lockbox_attribute[:decryption_key].nil?
+
+                  # it is possible that the encrypted attribute is not loaded, eg.
+                  # if the record was fetched partially (`User.select(:id).first`).
+                  # accessing a not loaded attribute raises an `ActiveModel::MissingAttributeError`.
+                  send(lockbox_attribute[:attribute]) if has_attribute?(lockbox_attribute[:encrypted_attribute])
                 end
                 super
               end
@@ -104,6 +110,49 @@ module Lockbox
                   end
                 end
               end
+
+              def update_columns(attributes)
+                return super unless attributes.is_a?(Hash)
+
+                # transform keys like Active Record
+                attributes = attributes.transform_keys do |key|
+                  n = key.to_s
+                  self.class.attribute_aliases[n] || n
+                end
+
+                lockbox_attributes = self.class.lockbox_attributes.slice(*attributes.keys.map(&:to_sym))
+                return super unless lockbox_attributes.any?
+
+                attributes_to_set = {}
+
+                lockbox_attributes.each do |key, lockbox_attribute|
+                  attribute = key.to_s
+                  # check read only
+                  verify_readonly_attribute(attribute)
+
+                  message = attributes[attribute]
+                  attributes.delete(attribute) unless lockbox_attribute[:migrating]
+                  encrypted_attribute = lockbox_attribute[:encrypted_attribute]
+                  ciphertext = self.class.send("generate_#{encrypted_attribute}", message, context: self)
+                  attributes[encrypted_attribute] = ciphertext
+                  attributes_to_set[attribute] = message
+                  attributes_to_set[lockbox_attribute[:attribute]] = message if lockbox_attribute[:migrating]
+                end
+
+                result = super(attributes)
+
+                # same logic as Active Record
+                # (although this happens before saving)
+                attributes_to_set.each do |k, v|
+                  if respond_to?(:write_attribute_without_type_cast, true)
+                    write_attribute_without_type_cast(k, v)
+                  else
+                    raw_write_attribute(k, v)
+                  end
+                end
+
+                result
+              end
             else
               def reload
                 self.class.lockbox_attributes.each do |_, v|
@@ -143,6 +192,10 @@ module Lockbox
               # however, we can try to use the original type if its already defined
               if attributes_to_define_after_schema_loads.key?(original_name.to_s)
                 attribute name, attributes_to_define_after_schema_loads[original_name.to_s].first
+              elsif options[:migrating]
+                # we use the original attribute for serialization in the encrypt and decrypt methods
+                # so we can use a generic value here
+                attribute name, ActiveRecord::Type::Value.new
               else
                 attribute name, :string
               end
@@ -213,12 +266,13 @@ module Lockbox
           define_method("#{name}=") do |message|
             # decrypt first for dirty tracking
             # don't raise error if can't decrypt previous
-            begin
-              send(name)
-            rescue Lockbox::DecryptionError
-              # this is expected for hybrid cryptography
-              warn "[lockbox] Decrypting previous value failed" unless options[:algorithm] == "hybrid"
-              nil
+            # don't try to decrypt if no decryption key given
+            unless options[:algorithm] == "hybrid" && options[:decryption_key].nil?
+              begin
+                send(name)
+              rescue Lockbox::DecryptionError
+                warn "[lockbox] Decrypting previous value failed"
+              end
             end
 
             send("lockbox_direct_#{name}=", message)
@@ -267,7 +321,7 @@ module Lockbox
                 # cache
                 # decrypt method does type casting
                 if respond_to?(:write_attribute_without_type_cast, true)
-                  write_attribute_without_type_cast(name, message) if !@attributes.frozen?
+                  write_attribute_without_type_cast(name.to_s, message) if !@attributes.frozen?
                 else
                   raw_write_attribute(name, message) if !@attributes.frozen?
                 end
@@ -316,7 +370,8 @@ module Lockbox
                 # do nothing
                 # encrypt will convert to binary
               else
-                type = (try(:attribute_types) || {})[name.to_s]
+                # use original name for serialized attributes
+                type = (try(:attribute_types) || {})[original_name.to_s]
                 message = type.serialize(message) if type
               end
             end
@@ -358,7 +413,8 @@ module Lockbox
                 # do nothing
                 # decrypt returns binary string
               else
-                type = (try(:attribute_types) || {})[name.to_s]
+                # use original name for serialized attributes
+                type = (try(:attribute_types) || {})[original_name.to_s]
                 message = type.deserialize(message) if type
                 message.force_encoding(Encoding::UTF_8) if !type || type.is_a?(ActiveModel::Type::String)
               end

data/lib/lockbox/utils.rb

@@ -4,22 +4,26 @@ module Lockbox
       options = options.except(:attribute, :encrypted_attribute, :migrating, :attached, :type)
       options[:encode] = false unless options.key?(:encode)
       options.each do |k, v|
-        if v.is_a?(Proc)
-          options[k] = context.instance_exec(&v) if v.respond_to?(:call)
+        if v.respond_to?(:call)
+          # context not present for pluck
+          # still possible to use if not dependent on context
+          options[k] = context ? context.instance_exec(&v) : v.call
         elsif v.is_a?(Symbol)
+          # context not present for pluck
+          raise Error, "Not available since :#{k} depends on record" unless context
           options[k] = context.send(v)
         end
       end
 
       unless options[:key] || options[:encryption_key] || options[:decryption_key]
-        options[:key] = Lockbox.attribute_key(table: table, attribute: attribute, master_key: options.delete(:master_key))
+        options[:key] = Lockbox.attribute_key(table: table, attribute: attribute, master_key: options.delete(:master_key), encode: false)
       end
 
       if options[:previous_versions].is_a?(Array)
         options[:previous_versions] = options[:previous_versions].dup
         options[:previous_versions].each_with_index do |version, i|
           if !(version[:key] || version[:encryption_key] || version[:decryption_key]) && version[:master_key]
-            options[:previous_versions][i] = version.merge(key: Lockbox.attribute_key(table: table, attribute: attribute, master_key: version.delete(:master_key)))
+            options[:previous_versions][i] = version.merge(key: Lockbox.attribute_key(table: table, attribute: attribute, master_key: version.delete(:master_key), encode: false))
           end
         end
       end
@@ -31,13 +35,13 @@ module Lockbox
       record.class.respond_to?(:lockbox_attachments) ? record.class.lockbox_attachments[name.to_sym] : nil
     end
 
-    def self.decode_key(key, size: 32)
+    def self.decode_key(key, size: 32, name: "Key")
       if key.encoding != Encoding::BINARY && key =~ /\A[0-9a-f]{#{size * 2}}\z/i
         key = [key].pack("H*")
       end
 
-      raise Lockbox::Error, "Key must use binary encoding" if key.encoding != Encoding::BINARY
-      raise Lockbox::Error, "Key must be 32 bytes" if key.bytesize != size
+      raise Lockbox::Error, "#{name} must be 32 bytes (64 hex digits)" if key.bytesize != size
+      raise Lockbox::Error, "#{name} must use binary encoding" if key.encoding != Encoding::BINARY
 
       key
     end

data/lib/lockbox/version.rb

@@ -1,3 +1,3 @@
 module Lockbox
-  VERSION = "0.4.2"
+  VERSION = "0.4.7"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: lockbox
 version: !ruby/object:Gem::Version
-  version: 0.4.2
+  version: 0.4.7
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-05-11 00:00:00.000000000 Z
+date: 2020-08-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -173,6 +173,7 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- SECURITY.md
 - lib/generators/lockbox/audits_generator.rb
 - lib/generators/lockbox/templates/migration.rb.tt
 - lib/generators/lockbox/templates/model.rb.tt
@@ -180,6 +181,7 @@ files:
 - lib/lockbox/active_storage_extensions.rb
 - lib/lockbox/aes_gcm.rb
 - lib/lockbox/box.rb
+- lib/lockbox/calculations.rb
 - lib/lockbox/carrier_wave_extensions.rb
 - lib/lockbox/encryptor.rb
 - lib/lockbox/io.rb