blind_index 2.0.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f656fa1765df9bf2bcfa9d994ccb9b5cf0504f27f7524d159323126462d973e0
-  data.tar.gz: 88d5f2cd786f840e75540a204dbe7fbb74794de954978d90cdc69297078cf752
+  metadata.gz: 80c561a0a96707de1176ae314d4d884e03390de9b8c4e23049d3649d7576937e
+  data.tar.gz: ea819c68b4d1a44c492799225d3250a86186cca1a355a3ab2e65a02069ae4062
 SHA512:
-  metadata.gz: 139d3d8f3aca413d0ee5045fe3212e6ed3327cdb6d0c60cb7eb2b314b4eb849abb42dcf17e386e23fb5521db2875b6c502b143fc46dc3305ad38151688b610be
-  data.tar.gz: 0f5df7f99a1b79f2eab0bc9ff959b9fd21c238e772e09265ef88690678134f539968d9aebe508b88e9881657563013cd53031947d6a6322e08c7e29d95f45902
+  metadata.gz: 44835258443127734b6940287e1768935884c86b53d90bb5c39a9a5db372b937649f81e5a6f16d9e4605cb78b19824759b019461491176cb4b2e0bfd1330858d
+  data.tar.gz: ddeeb0f625335d49e86ab0a5ff2350a7ae8c5b74ffd0322d7504763818b05ee6c648ea0d85b395c2f197266adfcb18015b37e556d6cbee04aea32f943d0123cd

data/CHANGELOG.md

@@ -1,4 +1,26 @@
-## 2.0.0 (2019-02-10)
+## 2.2.0 (2020-09-07)
+
+- Added support for `where` with table in Active Record 5.2+
+
+## 2.1.1 (2020-08-14)
+
+- Fixed `version` option
+
+## 2.1.0 (2020-07-06)
+
+- Improved performance of uniqueness validations
+- Fixed deprecation warnings in Ruby 2.7 with Mongoid
+
+## 2.0.2 (2020-06-01)
+
+- Improved error message for bad key length
+- Fixed `backfill` method with relations for Mongoid
+
+## 2.0.1 (2020-02-14)
+
+- Added `BlindIndex.backfill` method
+
+## 2.0.0 (2020-02-10)
 
 - Blind indexes are updated immediately instead of in a `before_validation` callback
 - Better Lockbox integration - no need to generate a separate key

data/README.md

@@ -10,7 +10,7 @@ Learn more about [securing sensitive data in Rails](https://ankane.org/sensitive
 
 ## How It Works
 
-We use [this approach](https://paragonie.com/blog/2017/05/building-searchable-encrypted-databases-with-php-and-sql) by Scott Arciszewski. To summarize, we compute a keyed hash of the sensitive data and store it in a column. To query, we apply the keyed hash function to the value we’re searching and then perform a database search. This results in performant queries for exact matches. `LIKE` queries are not possible, but you can index expressions.
+We use [this approach](https://paragonie.com/blog/2017/05/building-searchable-encrypted-databases-with-php-and-sql) by Scott Arciszewski. To summarize, we compute a keyed hash of the sensitive data and store it in a column. To query, we apply the keyed hash function to the value we’re searching and then perform a database search. This results in performant queries for exact matches. Efficient `LIKE` queries are [not possible](#like-ilike-and-full-text-searching), but you can index expressions.
 
 ## Leakage
 
@@ -26,13 +26,13 @@ Add this line to your application’s Gemfile:
 gem 'blind_index'
 ```
 
-## Getting Started
+## Prep
 
 Your model should already be set up with Lockbox or attr_encrypted. The examples are for a `User` model with `encrypts :email` or `attr_encrypted :email`. See the full examples for [Lockbox](https://ankane.org/securing-user-emails-lockbox) and [attr_encrypted](https://ankane.org/securing-user-emails-in-rails) if needed.
 
 Also, if you use attr_encrypted, [generate a key](#key-generation).
 
----
+## Getting Started
 
 Create a migration to add a column for the blind index
 
@@ -60,10 +60,7 @@ end
 Backfill existing records
 
 ```ruby
-User.unscoped.where(email_bidx: nil).find_each do |user|
-  user.compute_email_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User)
 ```
 
 And query away
@@ -72,9 +69,19 @@ And query away
 User.where(email: "test@example.org")
 ```
 
+## Expressions
+
+You can apply expressions to attributes before indexing and searching. This gives you the the ability to perform case-insensitive searches and more.
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, expression: ->(v) { v.downcase }
+end
+```
+
 ## Validations
 
-To prevent duplicates, use:
+You can use blind indexes for uniqueness validations.
 
 ```ruby
 class User < ApplicationRecord
@@ -82,15 +89,27 @@ class User < ApplicationRecord
 end
 ```
 
-We also recommend adding a unique index to the blind index column through a database migration.
+We recommend adding a unique index to the blind index column through a database migration.
 
-## Expressions
+```ruby
+add_index :users, :email_bidx, unique: true
+```
 
-You can apply expressions to attributes before indexing and searching. This gives you the the ability to perform case-insensitive searches and more.
+For `allow_blank: true`, use:
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, expression: ->(v) { v.presence }
+  validates :email, uniqueness: {allow_blank: true}
+end
+```
+
+For `case_sensitive: false`, use:
 
 ```ruby
 class User < ApplicationRecord
   blind_index :email, expression: ->(v) { v.downcase }
+  validates :email, uniqueness: true # for best performance, leave out {case_sensitive: false}
 end
 ```
 
@@ -115,10 +134,7 @@ end
 Backfill existing records
 
 ```ruby
-User.unscoped.where(email_ci_bidx: nil).find_each do |user|
-  user.compute_email_ci_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User, columns: [:email_ci_bidx])
 ```
 
 And query away
@@ -168,10 +184,7 @@ end
 This allows you to backfill records while still querying the unencrypted field.
 
 ```ruby
-User.unscoped.where(email_bidx: nil).find_each do |user|
-  user.compute_migrated_email_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User)
 ```
 
 Once that completes, you can remove the `migrating` option.
@@ -196,10 +209,7 @@ end
 This will keep the new column synced going forward. Next, backfill the data:
 
 ```ruby
-User.unscoped.where(email_bidx_v2: nil).find_each do |user|
-  user.compute_rotated_email_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User, columns: [:email_bidx_v2])
 ```
 
 Then update your model
@@ -279,6 +289,30 @@ or create `config/initializers/blind_index.rb` with something like
 BlindIndex.master_key = Rails.application.credentials.blind_index_master_key
 ```
 
+## LIKE, ILIKE, and Full-Text Searching
+
+Unfortunately, blind indexes can’t be used for `LIKE`, `ILIKE`, or full-text searching. Instead, records must be loaded, decrypted, and searched in memory.
+
+For `LIKE`, use:
+
+```ruby
+User.select { |u| u.email.include?("value") }
+```
+
+For `ILIKE`, use:
+
+```ruby
+User.select { |u| u.email =~ /value/i }
+```
+
+For full-text or fuzzy searching, use a gem like [FuzzyMatch](https://github.com/seamusabshere/fuzzy_match):
+
+```ruby
+FuzzyMatch.new(User.all, read: :email).find("value")
+```
+
+If the number of records is large, try to find a way to narrow it down. An [expression index](#expressions) is one way to do this, but leaks which records have the same value of the expression, so use it carefully.
+
 ## Reference
 
 Set default options in an initializer with:
@@ -448,3 +482,5 @@ cd blind_index
 bundle install
 bundle exec rake test
 ```
+
+For security issues, send an email to the address on [this page](https://github.com/ankane).

data/lib/blind_index.rb

@@ -4,6 +4,7 @@ require "openssl"
 require "argon2/kdf"
 
 # modules
+require "blind_index/backfill"
 require "blind_index/key_generator"
 require "blind_index/model"
 require "blind_index/version"
@@ -116,17 +117,21 @@ module BlindIndex
     key
   end
 
-  def self.decode_key(key)
+  def self.decode_key(key, name: "Key")
     # decode hex key
     if key.encoding != Encoding::BINARY && key =~ /\A[0-9a-f]{64}\z/i
       key = [key].pack("H*")
     end
 
-    raise BlindIndex::Error, "Key must use binary encoding" if key.encoding != Encoding::BINARY
-    raise BlindIndex::Error, "Key must be 32 bytes" if key.bytesize != 32
+    raise BlindIndex::Error, "#{name} must be 32 bytes (64 hex digits)" if key.bytesize != 32
+    raise BlindIndex::Error, "#{name} must use binary encoding" if key.encoding != Encoding::BINARY
 
     key
   end
+
+  def self.backfill(relation, columns: nil, batch_size: 1000)
+    Backfill.new(relation, columns: columns, batch_size: batch_size).perform
+  end
 end
 
 ActiveSupport.on_load(:active_record) do
@@ -136,17 +141,18 @@ ActiveSupport.on_load(:active_record) do
   ActiveRecord::TableMetadata.prepend(BlindIndex::Extensions::TableMetadata)
   ActiveRecord::DynamicMatchers::Method.prepend(BlindIndex::Extensions::DynamicMatchers)
 
-  unless ActiveRecord::VERSION::STRING.start_with?("5.1.")
+  unless ActiveRecord::VERSION::STRING.to_f == 5.1
     ActiveRecord::Validations::UniquenessValidator.prepend(BlindIndex::Extensions::UniquenessValidator)
   end
-end
 
-if defined?(Mongoid)
-  # TODO find better ActiveModel hook
-  require "active_model/callbacks"
-  ActiveModel::Callbacks.include(BlindIndex::Model)
+  if ActiveRecord::VERSION::STRING.to_f >= 5.2
+    ActiveRecord::PredicateBuilder.prepend(BlindIndex::Extensions::PredicateBuilder)
+  end
+end
 
+ActiveSupport.on_load(:mongoid) do
   require "blind_index/mongoid"
+  Mongoid::Document::ClassMethods.include(BlindIndex::Model)
   Mongoid::Criteria.prepend(BlindIndex::Mongoid::Criteria)
   Mongoid::Validatable::UniquenessValidator.prepend(BlindIndex::Mongoid::UniquenessValidator)
 end

data/lib/blind_index/backfill.rb

@@ -0,0 +1,113 @@
+module BlindIndex
+  class Backfill
+    attr_reader :blind_indexes
+
+    def initialize(relation, batch_size:, columns:)
+      @relation = relation
+      @transaction = @relation.respond_to?(:transaction)
+      @batch_size = batch_size
+      @blind_indexes = @relation.blind_indexes
+      filter_columns!(columns) if columns
+    end
+
+    def perform
+      each_batch do |records|
+        backfill_records(records)
+      end
+    end
+
+    private
+
+    # modify in-place
+    def filter_columns!(columns)
+      columns = Array(columns).map(&:to_s)
+      blind_indexes.select! { |_, v| columns.include?(v[:bidx_attribute]) }
+      bad_columns = columns - blind_indexes.map { |_, v| v[:bidx_attribute] }
+      raise ArgumentError, "Bad column: #{bad_columns.first}" if bad_columns.any?
+    end
+
+    def build_relation
+      # build relation
+      relation = @relation
+
+      if defined?(ActiveRecord::Base) && relation.is_a?(ActiveRecord::Base)
+        relation = relation.unscoped
+      end
+
+      # convert from possible class to ActiveRecord::Relation or Mongoid::Criteria
+      relation = relation.all
+
+      attributes = blind_indexes.map { |_, v| v[:bidx_attribute] }
+
+      if defined?(ActiveRecord::Relation) && relation.is_a?(ActiveRecord::Relation)
+        base_relation = relation.unscoped
+        or_relation = relation.unscoped
+
+        attributes.each_with_index do |attribute, i|
+          or_relation =
+            if i == 0
+              base_relation.where(attribute => nil)
+            else
+              or_relation.or(base_relation.where(attribute => nil))
+            end
+        end
+
+        relation.merge(or_relation)
+      else
+        relation.merge(relation.unscoped.or(attributes.map { |a| {a => nil} }))
+      end
+    end
+
+    def each_batch
+      relation = build_relation
+
+      if relation.respond_to?(:find_in_batches)
+        relation.find_in_batches(batch_size: @batch_size) do |records|
+          yield records
+        end
+      else
+        # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
+        # use cursor for Mongoid
+        records = []
+        relation.all.each do |record|
+          records << record
+          if records.length == @batch_size
+            yield records
+            records = []
+          end
+        end
+        yield records if records.any?
+      end
+    end
+
+    def backfill_records(records)
+      # do expensive blind index computation outside of transaction
+      records.each do |record|
+        blind_indexes.each do |k, v|
+          record.send("compute_#{k}_bidx") if !record.send(v[:bidx_attribute])
+        end
+      end
+
+      # don't need to save records that went from nil => nil
+      records.select! { |r| r.changed? }
+
+      if records.any?
+        with_transaction do
+          records.each do |record|
+            record.save!(validate: false)
+          end
+        end
+      end
+    end
+
+    def with_transaction
+      if @transaction
+        @relation.transaction do
+          yield
+        end
+      else
+        yield
+      end
+    end
+  end
+end

data/lib/blind_index/extensions.rb

@@ -1,22 +1,24 @@
 module BlindIndex
   module Extensions
     module TableMetadata
-      def resolve_column_aliases(hash)
-        new_hash = super
-        if has_blind_indexes?
-          hash.each do |key, _|
-            if key.respond_to?(:to_sym) && (bi = klass.blind_indexes[key.to_sym]) && !new_hash[key].is_a?(ActiveRecord::StatementCache::Substitute)
-              value = new_hash.delete(key)
-              new_hash[bi[:bidx_attribute]] =
-                if value.is_a?(Array)
-                  value.map { |v| BlindIndex.generate_bidx(v, **bi) }
-                else
-                  BlindIndex.generate_bidx(value, **bi)
-                end
+      if ActiveRecord::VERSION::STRING.to_f < 5.2
+        def resolve_column_aliases(hash)
+          new_hash = super
+          if has_blind_indexes?
+            hash.each_key do |key|
+              if key.respond_to?(:to_sym) && (bi = klass.blind_indexes[key.to_sym]) && !new_hash[key].is_a?(ActiveRecord::StatementCache::Substitute)
+                value = new_hash.delete(key)
+                new_hash[bi[:bidx_attribute]] =
+                  if value.is_a?(Array)
+                    value.map { |v| BlindIndex.generate_bidx(v, **bi) }
+                  else
+                    BlindIndex.generate_bidx(value, **bi)
+                  end
+              end
             end
           end
+          new_hash
         end
-        new_hash
       end
 
       # memoize for performance
@@ -28,11 +30,36 @@ module BlindIndex
       end
     end
 
+    module PredicateBuilder
+      # https://github.com/rails/rails/commit/56f30962b84fc53b76001301fb830c1594fd377e
+      def build(attribute, value, *args)
+        if table.has_blind_indexes? && (bi = table.send(:klass).blind_indexes[attribute.name.to_sym]) && !value.is_a?(ActiveRecord::StatementCache::Substitute)
+          attribute = attribute.relation[bi[:bidx_attribute]]
+          value =
+            if value.is_a?(Array)
+              value.map { |v| BlindIndex.generate_bidx(v, **bi) }
+            else
+              BlindIndex.generate_bidx(value, **bi)
+            end
+        end
+
+        super(attribute, value, *args)
+      end
+    end
+
     module UniquenessValidator
-      if ActiveRecord::VERSION::STRING >= "5.2"
+      def validate_each(record, attribute, value)
+        klass = record.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
+          value = record.read_attribute_for_validation(bi[:bidx_attribute])
+        end
+        super(record, attribute, value)
+      end
+
+      # change attribute name here instead of validate_each for better error message
+      if ActiveRecord::VERSION::STRING.to_f >= 5.2
         def build_relation(klass, attribute, value)
           if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
-            value = BlindIndex.generate_bidx(value, **bi)
             attribute = bi[:bidx_attribute]
           end
           super(klass, attribute, value)
@@ -40,7 +67,6 @@ module BlindIndex
       else
         def build_relation(klass, table, attribute, value)
           if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
-            value = BlindIndex.generate_bidx(value, **bi)
             attribute = bi[:bidx_attribute]
           end
           super(klass, table, attribute, value)

data/lib/blind_index/key_generator.rb

@@ -11,7 +11,7 @@ module BlindIndex
       raise ArgumentError, "Missing field for key generation" if bidx_attribute.to_s.empty?
 
       c = "\x7E"*32
-      root_key = hkdf(BlindIndex.decode_key(@master_key), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
+      root_key = hkdf(BlindIndex.decode_key(@master_key, name: "Master key"), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
       hash_hmac("sha256", pack([table, bidx_attribute, bidx_attribute]), root_key)
     end
 

data/lib/blind_index/model.rb

@@ -10,7 +10,7 @@ module BlindIndex
         # check here so we validate rotate options as well
         unknown_keywords = options.keys - [:algorithm, :attribute, :bidx_attribute,
           :callback, :cost, :encode, :expression, :insecure_key, :iterations, :key,
-          :legacy, :master_key, :size, :slow]
+          :legacy, :master_key, :size, :slow, :version]
         raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
 
         attribute = options[:attribute] || name
@@ -65,7 +65,7 @@ module BlindIndex
           end
 
           define_method method_name do
-            self.send("#{bidx_attribute}=", self.class.send(class_method_name, send(attribute)))
+            send("#{bidx_attribute}=", self.class.send(class_method_name, send(attribute)))
           end
 
           if callback

data/lib/blind_index/mongoid.rb

@@ -26,9 +26,9 @@ module BlindIndex
 
               criterion[bidx_key] =
                 if value.is_a?(Array)
-                  value.map { |v| BlindIndex.generate_bidx(v, bi) }
+                  value.map { |v| BlindIndex.generate_bidx(v, **bi) }
                 else
-                  BlindIndex.generate_bidx(value, bi)
+                  BlindIndex.generate_bidx(value, **bi)
                 end
             end
           end
@@ -39,9 +39,18 @@ module BlindIndex
     end
 
     module UniquenessValidator
+      def validate_each(record, attribute, value)
+        klass = record.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
+          value = record.read_attribute_for_validation(bi[:bidx_attribute])
+        end
+        super(record, attribute, value)
+      end
+
+      # change attribute name here instead of validate_each for better error message
       def create_criteria(base, document, attribute, value)
-        if base.respond_to?(:blind_indexes) && (bi = base.blind_indexes[attribute])
-          value = BlindIndex.generate_bidx(value, bi)
+        klass = document.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
           attribute = bi[:bidx_attribute]
         end
         super(base, document, attribute, value)

data/lib/blind_index/version.rb

@@ -1,3 +1,3 @@
 module BlindIndex
-  VERSION = "2.0.0"
+  VERSION = "2.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: blind_index
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.2.0
 platform: ruby
 authors:
 - Andrew Kane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-02-10 00:00:00.000000000 Z
+date: 2020-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -174,6 +174,7 @@ files:
 - LICENSE.txt
 - README.md
 - lib/blind_index.rb
+- lib/blind_index/backfill.rb
 - lib/blind_index/extensions.rb
 - lib/blind_index/key_generator.rb
 - lib/blind_index/model.rb