daffy_lib 0.1.0

data/README.md

@@ -0,0 +1,116 @@
+[![CircleCI](https://circleci.com/gh/Zetatango/daffy_lib.svg?style=svg)](https://circleci.com/gh/Zetatango/daffy_lib) [![codecov](https://codecov.io/gh/Zetatango/daffy_lib/branch/master/graph/badge.svg?token=WxED9350q4)](https://codecov.io/gh/Zetatango/daffy_lib) [![Gem Version](https://badge.fury.io/rb/daffy_lib.svg)](https://badge.fury.io/rb/daffy_lib) 
+# DaffyLib
+
+This gem is a caching encryptor which improves performance when encrypting/decrypting large amounts of data.  It will keep a plaintext key cached for a given amount of time, as well as provide partitioning to allow entire rows to be encrypted with the same key.  Keys are uniquely identified by a pair of a partition guid and an encryption epoch.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'daffy_lib'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install daffy_lib
+
+## Usage
+
+We illustrate usage of this library with an example.  Suppose we have classes `User` and `User::Attribute` where attributes belong to users and have values that need to be encrypted.  We wish for `User` to be the partition provider for `User::Attribute`.
+
+The `User` class will need to `include DaffyLib::PartitionProvider` and implement the method `provider_partition_guid` which returns an identifier, for instance a user's `guid`.
+
+The `User::Attribute` class will need to `include DaffyLib::PartitionProvider` as well as `include DaffyLib::HasEncryptedAttributes`.  It will need to declare `partition_provider :user`.  
+
+It then needs to implement a `generate_partition_guid` which returns the linked `User`'s `guid`, as well as a `generate_encryption_epoch` method which defines the encryption epoch.  The suggested implementations are:
+
+  ```
+  def generate_partition_guid
+    return partition_guid if partition_guid.present?
+
+    self.partition_guid = provider_partition_guid
+  end
+
+  def generate_encryption_epoch
+    return encryption_epoch if encryption_epoch.present?
+
+    self.encryption_epoch = DaffyLib::KeyManagementService.encryption_key_epoch(Time.now)
+  end
+  ```
+
+Now suppose the `User::Attributes` has a `values` field to be encrypted.  One declares
+
+```
+attr_encrypted :values, encryptor: ZtCachingEncryptor, encrypt_method: :zt_encrypt, decrypt_method: :zt_decrypt,
+                        encode: true, partition_guid: proc { |object| object.generate_partition_guid },
+                        encryption_epoch: proc { |object| object.generate_encryption_epoch }, expires_in: 5.minutes
+```
+                                         
+where the `expires_in` field denotes how long a plaintext key should be kept in cache.
+
+Note further that a class can be its own partition provider; i.e. if `User` itself had encrypted attributes, all the steps above for `User::Attributes` apply, except there is no need to declare `partition_provider`, and the recommended implementation for `generate_partition_guid` is to return (or create) the `guid` of the `User`.
+
+There are partial rake tasks to assist with the necessary database migrations included.
+
+Run `rake db:migrate:add_encryption_keys_table` to generate the migration file to add the encryption keys table.  Add the following lines for indexing to the generated file.
+
+```
+t.index [:guid], name: :index_encryption_keys_on_guid, unique: true
+t.index [:partition_guid, :key_epoch], name: :index_encryption_keys, unique: true
+
+```
+Next, if the models do not have existing records, one can run `rake db:migrate:add_encryption_fields[modelname]` to add the `partition_guid` and `encryption_epoch` columns.
+
+However, if there may already be existing records, then the `generate_partition_guid` and `generate_encryption_epoch` methods need to be invoked before the new columns can be set to required.  Below is a sample migration file for our example above, where one should replace `models` with their own.
+```
+  def up
+    models = %i[users users/attributes]
+
+    models.each do |model|
+
+      model_classname = model.to_s.camelize.singularize.constantize
+      model_tablename = model.to_s.gsub('/', '_')
+
+      add_column model_tablename, :partition_guid, :string
+      add_column model_tablename, :encryption_epoch, :datetime
+
+      model_classname.reset_column_information
+      model_classname.find_each do |record|
+        record.generate_partition_guid
+        record.generate_encryption_epoch
+
+        record.save!(validate: false)
+      end
+
+      change_column model_tablename, :partition_guid, :string, null: false
+      change_column model_tablename, :encryption_epoch, :datetime, null: false
+    end
+  end
+
+  def down
+    models = %i[users users/attributes]
+
+    models.each do |model|
+      model_tablename = model.to_s.gsub('/', '_')
+
+      remove_column model_tablename, :partition_guid
+      remove_column model_tablename, :encryption_epoch
+    end
+  end
+end
+```
+
+
+## Development
+
+Development on this project should occur on separate feature branches and pull requests should be submitted. When submitting a pull request, the pull request comment template should be filled out as much as possible to ensure a quick review and increase the likelihood of the pull request being accepted.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/Zetatango/daffy_lib.
+

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/SECURITY.MD

@@ -0,0 +1,5 @@
+# Security Policy
+
+## Reporting a Vulnerability
+
+For reporting confirmed or suspected vulnerabilities, please refer to https://www.arioplatform.com/security.

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "daffy_lib"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/daffy_lib.gemspec

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require_relative 'lib/daffy_lib/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "daffy_lib"
+  spec.version       = DaffyLib::VERSION
+  spec.authors       = ["Benoît Jeaurond, Weiyun Lu"]
+  spec.email         = ["weiyun.lu@arioplatform.com"]
+
+  spec.summary       = 'A library for caching encryptor'
+  spec.homepage      = 'https://github.com/Zetatango/daffy_lib'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'attr_encrypted'
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'bundler-audit'
+  spec.add_development_dependency 'codecov'
+  spec.add_development_dependency 'factory_bot_rails'
+  spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'rspec'
+  spec.add_development_dependency 'rspec-collection_matchers'
+  spec.add_development_dependency 'rspec-mocks'
+  spec.add_development_dependency 'rspec-rails'
+  spec.add_development_dependency 'rspec_junit_formatter'
+  spec.add_development_dependency 'rubocop'
+  spec.add_development_dependency 'rubocop-performance'
+  spec.add_development_dependency 'rubocop-rspec'
+  spec.add_development_dependency 'rubocop_runner'
+  spec.add_development_dependency 'simplecov'
+  spec.add_development_dependency 'sqlite3'
+  spec.add_development_dependency 'timecop'
+
+  spec.add_dependency 'porky_lib'
+  spec.add_dependency 'rails'
+  spec.add_dependency 'redis'
+end

data/lib/daffy_lib.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+require "daffy_lib/version"
+
+module DaffyLib
+  require 'daffy_lib/caching_encryptor'
+  require 'daffy_lib/concerns/has_guid'
+  require 'daffy_lib/concerns/has_encrypted_attributes'
+  require 'daffy_lib/concerns/partition_provider'
+  require 'daffy_lib/models/application_record'
+  require 'daffy_lib/models/encryption_key'
+  require 'daffy_lib/railtie' if defined?(Rails) # for the rake tasks
+  require 'daffy_lib/services/key_management_service'
+  require 'daffy_lib/validators/string_validator'
+end

data/lib/daffy_lib/caching_encryptor.rb

@@ -0,0 +1,114 @@
+# frozen_string_literal: true
+
+class DaffyLib::CachingEncryptor
+  class CachingEncryptorException < StandardError; end
+
+  class EncryptionFailedException < CachingEncryptorException; end
+  class DecryptionFailedException < CachingEncryptorException; end
+  class InvalidParameterException < CachingEncryptorException; end
+
+  def self.zt_encrypt(*args, &_block)
+    data, partition_guid, encryption_epoch, expires_in, cmk_key_id = validate_encrypt_params(*args)
+
+    kms = DaffyLib::KeyManagementService.new(partition_guid, expires_in, cmk_key_id)
+
+    key_info = kms.find_or_create_encryption_key(encryption_epoch)
+
+    plaintext_key = kms.retrieve_plaintext_key(key_info)
+
+    encryption_result = PorkyLib::Symmetric.instance.encrypt_with_key(data, plaintext_key)
+
+    # The value returned from this method is stored in the encrypted_{attr} field in the DB, but there isn't a way to tell the attr_encrypted library
+    # the value of the nonce/IV to store or the value of the encryption key to store. As a result, we will store a JSON object as the encrypted_{attr},
+    # with the raw byte values Base64 encoded.
+    {
+      key_guid: key_info.guid,
+      # Store this with the data in case we need to decrypt outside the platform
+      key: key_info.encrypted_data_encryption_key,
+      data: Base64.encode64(encryption_result.ciphertext),
+      nonce: Base64.encode64(encryption_result.nonce)
+    }.to_json
+  rescue DaffyLib::KeyManagementService::KeyManagementServiceException => e
+    Rails.logger.error("KeyManagementService exception on encrypt: #{e.message}")
+
+    raise EncryptionFailedException
+  rescue RbNaCl::CryptoError, RbNaCl::LengthError => e
+    Rails.logger.error("RbNaCl exception on encrypt: #{e.message}")
+
+    raise EncryptionFailedException
+  end
+
+  def self.zt_decrypt(*args, &block)
+    value, expires_in, cmk_key_id = validate_decrypt_params(*args)
+
+    ciphertext_info = JSON.parse(value, symbolize_names: true)
+
+    # Call the legacy decrypt function if there is no key_guid present
+    return legacy_decrypt(*args, block) unless ciphertext_info.key?(:key_guid)
+
+    key_guid = ciphertext_info[:key_guid]
+    ciphertext = Base64.decode64(ciphertext_info[:data])
+    nonce = Base64.decode64(ciphertext_info[:nonce])
+
+    key_info = DaffyLib::EncryptionKey.find_by!(guid: key_guid)
+
+    kms = DaffyLib::KeyManagementService.new(key_info.partition_guid, expires_in, cmk_key_id)
+    plaintext_key = kms.retrieve_plaintext_key(key_info)
+
+    PorkyLib::Symmetric.instance.decrypt_with_key(
+      ciphertext,
+      plaintext_key,
+      nonce
+    ).plaintext
+  rescue JSON::JSONError => e
+    Rails.logger.error("JSON parse error on decryption: #{e.message}")
+
+    raise DecryptionFailedException
+  rescue ActiveRecord::RecordNotFound => e
+    Rails.logger.error("Failed to find encryption key for guid #{key_guid} on decrypt: #{e.message}")
+
+    raise DecryptionFailedException
+  rescue DaffyLib::KeyManagementService::KeyManagementServiceException => e
+    Rails.logger.error("KeyManagementService exception on decrypt: #{e.message}")
+
+    raise DecryptionFailedException
+  rescue RbNaCl::CryptoError, RbNaCl::LengthError => e
+    Rails.logger.error("RbNaCl exception on decrypt: #{e.message}")
+
+    raise DecryptionFailedException
+  end
+
+  def self.legacy_decrypt(*args, &_block)
+    ciphertext_data = JSON.parse(args.first[:value], symbolize_names: true)
+    ciphertext_key = Base64.decode64(ciphertext_data[:key])
+    ciphertext = Base64.decode64(ciphertext_data[:data])
+    nonce = Base64.decode64(ciphertext_data[:nonce])
+
+    PorkyLib::Symmetric.instance.decrypt(ciphertext_key, ciphertext, nonce).first
+  end
+  private_class_method :legacy_decrypt
+
+  def self.validate_encrypt_params(*args)
+    data = args.first[:value]
+    partition_guid = args.first[:partition_guid]
+    encryption_epoch = args.first[:encryption_epoch]
+    expires_in = args.first[:expires_in]
+    cmk_key_id = args.first[:cmk_key_id]
+
+    raise InvalidParameterException unless data.present? && partition_guid.present? && encryption_epoch.present? && expires_in.present? && cmk_key_id.present?
+
+    [data, partition_guid, encryption_epoch, expires_in, cmk_key_id]
+  end
+  private_class_method :validate_encrypt_params
+
+  def self.validate_decrypt_params(*args)
+    value = args.first[:value]
+    expires_in = args.first[:expires_in]
+    cmk_key_id = args.first[:cmk_key_id]
+
+    raise InvalidParameterException unless value.present? && expires_in.present? && cmk_key_id.present?
+
+    [value, expires_in, cmk_key_id]
+  end
+  private_class_method :validate_decrypt_params
+end

data/lib/daffy_lib/concerns/has_encrypted_attributes.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module DaffyLib::HasEncryptedAttributes
+  extend ActiveSupport::Concern
+
+  included do
+    before_create :generate_partition_guid, :generate_encryption_epoch
+
+    validate :ensure_encryption_info_does_not_change
+  end
+
+  def generate_partition_guid
+    raise NoMethodError
+  end
+
+  def generate_encryption_epoch
+    raise NoMethodError
+  end
+
+  private
+
+  def ensure_encryption_info_does_not_change
+    return if new_record?
+
+    errors.add(:partition_guid, 'cannot be changed for persisted records') if partition_guid_changed?
+    errors.add(:encryption_epoch, 'cannot be changed for persisted records') if encryption_epoch_changed?
+  end
+end

data/lib/daffy_lib/concerns/has_guid.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+require 'active_support'
+
+module DaffyLib::HasGuid
+  extend ActiveSupport::Concern
+
+  REGEXP = /^[\w]+_[\w]+$/.freeze
+
+  mattr_accessor :registry do
+    {}
+  end
+
+  class_methods do
+    # rubocop:disable Naming/PredicateName
+
+    def has_guid(prefix)
+      if DaffyLib::HasGuid.registry[prefix].present?
+        raise ArgumentError, "Prefix #{prefix} has already been registered by class #{DaffyLib::HasGuid.registry[prefix]}"
+      end
+
+      DaffyLib::HasGuid.registry[prefix] = self
+
+      class_eval do
+        cattr_accessor :has_guid_prefix do
+          prefix
+        end
+
+        before_create :generate_guid
+        validate :ensure_guid_does_not_change
+
+        def dup
+          super.tap { |duplicate| duplicate.guid = nil if duplicate.respond_to?('guid=') }
+        end
+
+        def to_param
+          guid.to_param
+        end
+
+        def self.generate_guid
+          "#{has_guid_prefix}_#{SecureRandom.base58(16)}"
+        end
+
+        def generate_guid
+          self.guid = guid.presence || self.class.generate_guid
+        end
+
+        def self.validation_regexp
+          /^#{has_guid_prefix}_[\w]+$/
+        end
+
+        def ensure_guid_does_not_change
+          return if new_record?
+          return unless guid_changed?
+
+          errors.add(:guid, 'cannot be changed for persisted records')
+        end
+      end
+    end
+    # rubocop:enable Naming/PredicateName
+  end
+end

data/lib/daffy_lib/concerns/partition_provider.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+module DaffyLib::PartitionProvider
+  extend ActiveSupport::Concern
+
+  class_methods do
+    def partition_provider(attribute)
+      class_eval do
+        cattr_accessor :partition_provider_attribute do
+          attribute
+        end
+
+        def provider_partition_guid
+          provider_info
+        end
+      end
+    end
+
+    def partition_provider_guid(guid_method, model)
+      class_eval do
+        cattr_accessor :model do
+          model
+        end
+
+        cattr_accessor :guid_method do
+          guid_method
+        end
+
+        def provider_partition_guid
+          provider_record_info(record)
+        end
+
+        private
+
+        def record
+          model.find_by(guid: send(guid_method))
+        end
+      end
+    end
+  end
+
+  def provider_info
+    raise ActiveRecord::RecordInvalid if partition_provider_attribute.nil?
+
+    provider_record_info(send(partition_provider_attribute))
+  end
+
+  private
+
+  def provider_record_info(record)
+    raise ActiveRecord::RecordInvalid unless record.present? && record.is_a?(DaffyLib::PartitionProvider)
+
+    info = record.send(:provider_partition_guid)
+
+    raise ActiveRecord::RecordInvalid if info.nil?
+
+    info
+  end
+end