blind_index 1.0.2 → 2.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2f39918c983140363af00fe1afeda800a7476a5727fd2c676c713b92222c1f35
-  data.tar.gz: b83995b7b33eddacb2988e7fdb8816a1b2fc644161114b5e191f3d9bb2de2bd2
+  metadata.gz: cca7398adad9e03afd11dd268f4e69d99abe9ef767d648525ceeaee02aa37ba2
+  data.tar.gz: 9ebc3c610a67aace050bffc6724cf7067f734bfe142ff2e0409cd73be66cbbb0
 SHA512:
-  metadata.gz: b2d03f17acbac80b5ecf8ac1f97321f3c68b0c689db1e50d95cc81cb3769086a5d8da6f8f0386325555c8eb715d46555281e00af38804cd240cadbefa45e86a4
-  data.tar.gz: 59092061d3b415d399f1a20232d680591ccc832af79387945d23d9309062bf10a4b8105062ea277af2fa06b454cd75f9ad3d49a44dabac1a990e2856bc700849
+  metadata.gz: d2bc3c733e12261e561cfb0e82a171c9c995079d20f032dba22f757b07dc23dddadbcc87a20e7c83f7a683f6c6904b2227235aee5e69523168d3df2cb50551ec
+  data.tar.gz: ccea3ba949b1643cbbe86f8a2c5fdbd8dc1837334d9db1055bd8cfc0b2f16ea00133029483ea876ddef1c44359f9dec8e1313cbfcd5f727c54f14e803b2110fe

data/CHANGELOG.md

@@ -1,3 +1,55 @@
+## 2.5.0 (2024-06-03)
+
+- Added support for Mongoid 9
+- Dropped support for Ruby < 3.1
+
+## 2.4.0 (2023-07-02)
+
+- Dropped support for Ruby < 3 and Rails < 6.1
+- Dropped support for Mongoid < 7
+
+## 2.3.2 (2023-04-26)
+
+- Added `key_table` and `key_attribute` options
+
+## 2.3.1 (2022-09-06)
+
+- Fixed error with `backfill` when `bidx_attribute` is a symbol
+
+## 2.3.0 (2022-01-16)
+
+- Added blind indexes to `filter_attributes`
+- Dropped support for Ruby < 2.6 and Rails < 5.2
+
+## 2.2.0 (2020-09-07)
+
+- Added support for `where` with table in Active Record 5.2+
+
+## 2.1.1 (2020-08-14)
+
+- Fixed `version` option
+
+## 2.1.0 (2020-07-06)
+
+- Improved performance of uniqueness validations
+- Fixed deprecation warnings in Ruby 2.7 with Mongoid
+
+## 2.0.2 (2020-06-01)
+
+- Improved error message for bad key length
+- Fixed `backfill` method with relations for Mongoid
+
+## 2.0.1 (2020-02-14)
+
+- Added `BlindIndex.backfill` method
+
+## 2.0.0 (2020-02-10)
+
+- Blind indexes are updated immediately instead of in a `before_validation` callback
+- Better Lockbox integration - no need to generate a separate key
+- The `argon2` gem has been replaced with `argon2-kdf` for less dependencies and Windows support
+- Removed deprecated `compute_email_bidx`
+
 ## 1.0.2 (2019-12-26)
 
 - Fixed `OpenSSL::KDF` error on some platforms
@@ -30,7 +82,7 @@ Breaking changes
 
 - Added `size` option
 - Added sanity checks for Argon2 cost parameters
-- Fixed ActiveRecord callback issues introduced in 0.3.3
+- Fixed Active Record callback issues introduced in 0.3.3
 
 ## 0.3.3 (2018-11-12)
 
@@ -59,13 +111,13 @@ Breaking changes
 
 ## 0.2.0 (2018-05-11)
 
-- Added support for ActiveRecord 4.2
+- Added support for Active Record 4.2
 - Improved validation support when multiple blind indexes
 - Fixed `nil` handling
 
 ## 0.1.1 (2018-04-09)
 
-- Added support for ActiveRecord 5.2
+- Added support for Active Record 5.2
 - Added `callback` option
 - Added support for `key` proc
 - Fixed error inheritance

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2017-2019 Andrew Kane
+Copyright (c) 2017-2023 Andrew Kane
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -6,11 +6,11 @@ Works with [Lockbox](https://github.com/ankane/lockbox) ([full example](https://
 
 Learn more about [securing sensitive data in Rails](https://ankane.org/sensitive-data-rails)
 
-[![Build Status](https://travis-ci.org/ankane/blind_index.svg?branch=master)](https://travis-ci.org/ankane/blind_index)
+[![Build Status](https://github.com/ankane/blind_index/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/blind_index/actions)
 
 ## How It Works
 
-We use [this approach](https://paragonie.com/blog/2017/05/building-searchable-encrypted-databases-with-php-and-sql) by Scott Arciszewski. To summarize, we compute a keyed hash of the sensitive data and store it in a column. To query, we apply the keyed hash function to the value we’re searching and then perform a database search. This results in performant queries for exact matches. `LIKE` queries are not possible, but you can index expressions.
+We use [this approach](https://paragonie.com/blog/2017/05/building-searchable-encrypted-databases-with-php-and-sql) by Scott Arciszewski. To summarize, we compute a keyed hash of the sensitive data and store it in a column. To query, we apply the keyed hash function to the value we’re searching and then perform a database search. This results in performant queries for exact matches. Efficient `LIKE` queries are [not possible](#like-ilike-and-full-text-searching), but you can index expressions.
 
 ## Leakage
 
@@ -23,41 +23,17 @@ Here’s a [great article](https://blog.cryptographyengineering.com/2019/02/11/a
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'blind_index'
+gem "blind_index"
 ```
 
-On Windows, also add:
+## Prep
 
-```ruby
-gem 'argon2', git: 'https://github.com/technion/ruby-argon2.git', submodules: true
-```
+Your model should already be set up with Lockbox or attr_encrypted. The examples are for a `User` model with `has_encrypted :email` or `attr_encrypted :email`. See the full examples for [Lockbox](https://ankane.org/securing-user-emails-lockbox) and [attr_encrypted](https://ankane.org/securing-user-emails-in-rails) if needed.
 
-Until `argon2 > 2.0.2` is released.
+Also, if you use attr_encrypted, [generate a key](#key-generation).
 
 ## Getting Started
 
-> Note: Your model should already be set up with Lockbox or attr_encrypted. The examples are for a `User` model with `encrypts :email` or `attr_encrypted :email`. See the full examples for [Lockbox](https://ankane.org/securing-user-emails-lockbox) and [attr_encrypted](https://ankane.org/securing-user-emails-in-rails) if needed.
-
-First, generate a key
-
-```ruby
-BlindIndex.generate_key
-```
-
-Store the key with your other secrets. This is typically Rails credentials or an environment variable ([dotenv](https://github.com/bkeepers/dotenv) is great for this). Be sure to use different keys in development and production. Keys don’t need to be hex-encoded, but it’s often easier to store them this way.
-
-Set the following environment variable with your key (you can use this one in development)
-
-```sh
-BLIND_INDEX_MASTER_KEY=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
-```
-
-or create `config/initializers/blind_index.rb` with something like
-
-```ruby
-BlindIndex.master_key = Rails.application.credentials.blind_index_master_key
-```
-
 Create a migration to add a column for the blind index
 
 ```ruby
@@ -84,10 +60,7 @@ end
 Backfill existing records
 
 ```ruby
-User.unscoped.where(email_bidx: nil).find_each do |user|
-  user.compute_email_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User)
 ```
 
 And query away
@@ -96,9 +69,19 @@ And query away
 User.where(email: "test@example.org")
 ```
 
+## Expressions
+
+You can apply expressions to attributes before indexing and searching. This gives you the the ability to perform case-insensitive searches and more.
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, expression: ->(v) { v.downcase }
+end
+```
+
 ## Validations
 
-To prevent duplicates, use:
+You can use blind indexes for uniqueness validations.
 
 ```ruby
 class User < ApplicationRecord
@@ -106,15 +89,27 @@ class User < ApplicationRecord
 end
 ```
 
-We also recommend adding a unique index to the blind index column through a database migration.
+We recommend adding a unique index to the blind index column through a database migration.
 
-## Expressions
+```ruby
+add_index :users, :email_bidx, unique: true
+```
 
-You can apply expressions to attributes before indexing and searching. This gives you the the ability to perform case-insensitive searches and more.
+For `allow_blank: true`, use:
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, expression: ->(v) { v.presence }
+  validates :email, uniqueness: {allow_blank: true}
+end
+```
+
+For `case_sensitive: false`, use:
 
 ```ruby
 class User < ApplicationRecord
   blind_index :email, expression: ->(v) { v.downcase }
+  validates :email, uniqueness: true # for best performance, leave out {case_sensitive: false}
 end
 ```
 
@@ -139,10 +134,7 @@ end
 Backfill existing records
 
 ```ruby
-User.unscoped.where(email_ci_bidx: nil).find_each do |user|
-  user.compute_email_ci_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User, columns: [:email_ci_bidx])
 ```
 
 And query away
@@ -169,18 +161,34 @@ You can also use virtual attributes to index data from multiple columns:
 ```ruby
 class User < ApplicationRecord
   attribute :initials, :string
+  blind_index :initials
 
-  # must come before the blind_index method so it runs first
   before_validation :set_initials, if: -> { changes.key?(:first_name) || changes.key?(:last_name) }
 
-  blind_index :initials
-
   def set_initials
     self.initials = "#{first_name[0]}#{last_name[0]}"
   end
 end
 ```
 
+## Migrating Data
+
+If you’re encrypting a column and adding a blind index at the same time, use the `migrating` option.
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, migrating: true
+end
+```
+
+This allows you to backfill records while still querying the unencrypted field.
+
+```ruby
+BlindIndex.backfill(User)
+```
+
+Once that completes, you can remove the `migrating` option.
+
 ## Key Rotation
 
 To rotate keys without downtime, add a new column:
@@ -201,10 +209,7 @@ end
 This will keep the new column synced going forward. Next, backfill the data:
 
 ```ruby
-User.unscoped.where(email_bidx_v2: nil).find_each do |user|
-  user.compute_rotated_email_bidx
-  user.save(validate: false)
-end
+BlindIndex.backfill(User, columns: [:email_bidx_v2])
 ```
 
 Then update your model
@@ -219,17 +224,27 @@ Finally, drop the old column.
 
 ## Key Separation
 
-The master key is used to generate unique keys for each blind index. This technique comes from [CipherSweet](https://ciphersweet.paragonie.com/internals/key-hierarchy). The table name and blind index column name are both used in this process. If you need to rename a table with blind indexes, or a blind index column itself, get the key:
+The master key is used to generate unique keys for each blind index. This technique comes from [CipherSweet](https://ciphersweet.paragonie.com/internals/key-hierarchy). The table name and blind index column name are both used in this process.
+
+You can get an individual key with:
 
 ```ruby
 BlindIndex.index_key(table: "users", bidx_attribute: "email_bidx")
 ```
 
-And set it directly before renaming:
+To rename a table with blind indexes, use:
 
 ```ruby
 class User < ApplicationRecord
-  blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"]
+  blind_index :email, key_table: "original_table"
+end
+```
+
+To rename a blind index column, use:
+
+```ruby
+class User < ApplicationRecord
+  blind_index :email, key_attribute: "original_column"
 end
 ```
 
@@ -257,148 +272,128 @@ For Mongoid, use:
 ```ruby
 class User
   field :email_bidx, type: String
+  index({email_bidx: 1})
 end
 ```
 
-## Reference
+## Key Generation
 
-Set default options in an initializer with:
-
-```ruby
-BlindIndex.default_options = {algorithm: :pbkdf2_sha256}
-```
+This is optional for Lockbox, as its master key is used by default.
 
-By default, blind indexes are encoded in Base64. Set a different encoding with:
+Generate a key with:
 
 ```ruby
-class User < ApplicationRecord
-  blind_index :email, encode: ->(v) { [v].pack("H*") }
-end
+BlindIndex.generate_key
 ```
 
-By default, blind indexes are 32 bytes. Set a smaller size with:
+Store the key with your other secrets. This is typically Rails credentials or an environment variable ([dotenv](https://github.com/bkeepers/dotenv) is great for this). Be sure to use different keys in development and production. Keys don’t need to be hex-encoded, but it’s often easier to store them this way.
 
-```ruby
-class User < ApplicationRecord
-  blind_index :email, size: 16
-end
+Set the following environment variable with your key (you can use this one in development)
+
+```sh
+BLIND_INDEX_MASTER_KEY=ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
 ```
 
-Set a key directly for an index with:
+or create `config/initializers/blind_index.rb` with something like
 
 ```ruby
-class User < ApplicationRecord
-  blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"]
-end
+BlindIndex.master_key = Rails.application.credentials.blind_index_master_key
 ```
 
-## Alternatives
-
-One alternative to blind indexing is to use a deterministic encryption scheme, like [AES-SIV](https://github.com/miscreant/miscreant). In this approach, the encrypted data will be the same for matches. We recommend blind indexing over deterministic encryption because:
-
-1. You can keep encryption consistent for all fields (both searchable and non-searchable)
-2. Blind indexing supports expressions
-
-## Upgrading
+## LIKE, ILIKE, and Full-Text Searching
 
-### 1.0.0
+Unfortunately, blind indexes can’t be used for `LIKE`, `ILIKE`, or full-text searching. Instead, records must be loaded, decrypted, and searched in memory.
 
-1.0.0 brings a number of improvements. Here are a few to be aware of:
+For `LIKE`, use:
 
-- Argon2id is the default algorithm for stronger security
-- You can use a master key instead of individual keys for each column
-- Columns no longer have an `encrypted_` prefix
+```ruby
+User.select { |u| u.email.include?("value") }
+```
 
-For existing fields, add:
+For `ILIKE`, use:
 
 ```ruby
-class User < ApplicationRecord
-  blind_index :email, legacy: true
-end
+User.select { |u| u.email =~ /value/i }
 ```
 
-#### Optional
-
-To rotate to new fields that use Argon2id and a master key, generate a master key:
+For full-text or fuzzy searching, use a gem like [FuzzyMatch](https://github.com/seamusabshere/fuzzy_match):
 
 ```ruby
-BlindIndex.generate_key
+FuzzyMatch.new(User.all, read: :email).find("value")
 ```
 
-And set `ENV["BLIND_INDEX_MASTER_KEY"]` or `BlindIndex.master_key`.
+If the number of records is large, try to find a way to narrow it down. An [expression index](#expressions) is one way to do this, but leaks which records have the same value of the expression, so use it carefully.
+
+## Reference
 
-Add a new column without the `encrypted_` prefix:
+Set default options in an initializer with:
 
 ```ruby
-add_column :users, :email_bidx, :string
-add_index :users, :email_bidx # unique: true if needed
+BlindIndex.default_options = {algorithm: :pbkdf2_sha256}
 ```
 
-And add to your model
+By default, blind indexes are encoded in Base64. Set a different encoding with:
 
 ```ruby
 class User < ApplicationRecord
-  blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"], legacy: true, rotate: {}
+  blind_index :email, encode: ->(v) { [v].pack("H*") }
 end
 ```
 
-> For more sensitive fields, use `rotate: {slow: true}`
-
-This will keep the new column synced going forward. Next, backfill the data:
+By default, blind indexes are 32 bytes. Set a smaller size with:
 
 ```ruby
-User.unscoped.where(email_bidx: nil).find_each do |user|
-  user.compute_rotated_email_bidx
-  user.save(validate: false)
+class User < ApplicationRecord
+  blind_index :email, size: 16
 end
 ```
 
-Then update your model
+Set a key directly for an index with:
 
 ```ruby
 class User < ApplicationRecord
-  blind_index :email
+  blind_index :email, key: ENV["USER_EMAIL_BLIND_INDEX_KEY"]
 end
 ```
 
-> For more sensitive fields, add `slow: true`
-
-Finally, drop the old column.
+## Compatibility
 
-### 0.3.0
+You can generate blind indexes from other languages as well. For Python, you can use [argon2-cffi](https://github.com/hynek/argon2-cffi).
 
-This version introduces a breaking change to enforce secure key generation. An error is thrown if your blind index key isn’t both binary and 32 bytes.
+```python
+from argon2.low_level import Type, hash_secret_raw
+from base64 import b64encode
 
-We recommend rotating your key if it doesn’t meet this criteria. You can generate a new key in the Rails console with:
+key = '289737bab72fa97b1f4b081cef00d7b7d75034bcf3183c363feaf3e6441777bc'
+value = 'test@example.org'
 
-```ruby
-SecureRandom.hex(32)
+bidx = b64encode(hash_secret_raw(
+    secret=value.encode(),
+    salt=bytes.fromhex(key),
+    time_cost=3,
+    memory_cost=2**12,
+    parallelism=1,
+    hash_len=32,
+    type=Type.ID
+))
 ```
 
-Update your model to convert the hex key to binary.
+## Alternatives
 
-```ruby
-class User < ApplicationRecord
-  blind_index :email, key: [ENV["USER_EMAIL_BLIND_INDEX_KEY"]].pack("H*")
-end
-```
+One alternative to blind indexing is to use a deterministic encryption scheme, like [AES-SIV](https://github.com/miscreant/miscreant). In this approach, the encrypted data will be the same for matches. We recommend blind indexing over deterministic encryption because:
 
-And recompute the blind index.
+1. You can keep encryption consistent for all fields (both searchable and non-searchable)
+2. Blind indexing supports expressions
 
-```ruby
-User.unscoped.find_each do |user|
-  user.compute_email_bidx
-  user.save(validate: false)
-end
-```
+## Upgrading
 
-To continue without rotating, set:
+### 2.0.0
 
-```ruby
-class User < ApplicationRecord
-  blind_index :email, insecure_key: true
-end
-```
+2.0.0 brings a number of improvements.
+
+- Blind indexes are updated immediately instead of in a `before_validation` callback
+- Better Lockbox integration - no need to generate a separate key
+- There’s a new gem for Argon2 that has no dependencies and (officially) supports Windows
 
 ## History
 
@@ -421,3 +416,5 @@ cd blind_index
 bundle install
 bundle exec rake test
 ```
+
+For security issues, send an email to the address on [this page](https://github.com/ankane).

data/lib/blind_index/backfill.rb

@@ -0,0 +1,117 @@
+module BlindIndex
+  class Backfill
+    attr_reader :blind_indexes
+
+    def initialize(relation, batch_size:, columns:)
+      @relation = relation
+      @transaction = @relation.respond_to?(:transaction) && !mongoid_relation?(relation.all)
+      @batch_size = batch_size
+      @blind_indexes = @relation.blind_indexes
+      filter_columns!(columns) if columns
+    end
+
+    def perform
+      each_batch do |records|
+        backfill_records(records)
+      end
+    end
+
+    private
+
+    # modify in-place
+    def filter_columns!(columns)
+      columns = Array(columns).map(&:to_s)
+      blind_indexes.select! { |_, v| columns.include?(v[:bidx_attribute].to_s) }
+      bad_columns = columns - blind_indexes.map { |_, v| v[:bidx_attribute].to_s }
+      raise ArgumentError, "Bad column: #{bad_columns.first}" if bad_columns.any?
+    end
+
+    def build_relation
+      # build relation
+      relation = @relation
+
+      if defined?(ActiveRecord::Base) && relation.is_a?(ActiveRecord::Base)
+        relation = relation.unscoped
+      end
+
+      # convert from possible class to ActiveRecord::Relation or Mongoid::Criteria
+      relation = relation.all
+
+      attributes = blind_indexes.map { |_, v| v[:bidx_attribute] }
+
+      if defined?(ActiveRecord::Relation) && relation.is_a?(ActiveRecord::Relation)
+        base_relation = relation.unscoped
+        or_relation = relation.unscoped
+
+        attributes.each_with_index do |attribute, i|
+          or_relation =
+            if i == 0
+              base_relation.where(attribute => nil)
+            else
+              or_relation.or(base_relation.where(attribute => nil))
+            end
+        end
+
+        relation.merge(or_relation)
+      else
+        relation.merge(relation.unscoped.or(attributes.map { |a| {a => nil} }))
+      end
+    end
+
+    def each_batch
+      relation = build_relation
+
+      if relation.respond_to?(:find_in_batches)
+        relation.find_in_batches(batch_size: @batch_size) do |records|
+          yield records
+        end
+      else
+        # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb
+        # use cursor for Mongoid
+        records = []
+        relation.all.each do |record|
+          records << record
+          if records.length == @batch_size
+            yield records
+            records = []
+          end
+        end
+        yield records if records.any?
+      end
+    end
+
+    def backfill_records(records)
+      # do expensive blind index computation outside of transaction
+      records.each do |record|
+        blind_indexes.each do |k, v|
+          record.send("compute_#{k}_bidx") if !record.send(v[:bidx_attribute])
+        end
+      end
+
+      # don't need to save records that went from nil => nil
+      records.select! { |r| r.changed? }
+
+      if records.any?
+        with_transaction do
+          records.each do |record|
+            record.save!(validate: false)
+          end
+        end
+      end
+    end
+
+    def mongoid_relation?(relation)
+      defined?(Mongoid::Criteria) && relation.is_a?(Mongoid::Criteria)
+    end
+
+    def with_transaction
+      if @transaction
+        @relation.transaction do
+          yield
+        end
+      else
+        yield
+      end
+    end
+  end
+end

data/lib/blind_index/extensions.rb

@@ -1,25 +1,6 @@
 module BlindIndex
   module Extensions
-    # ActiveRecord 5.0+
     module TableMetadata
-      def resolve_column_aliases(hash)
-        new_hash = super
-        if has_blind_indexes?
-          hash.each do |key, _|
-            if key.respond_to?(:to_sym) && (bi = klass.blind_indexes[key.to_sym]) && !new_hash[key].is_a?(ActiveRecord::StatementCache::Substitute)
-              value = new_hash.delete(key)
-              new_hash[bi[:bidx_attribute]] =
-                if value.is_a?(Array)
-                  value.map { |v| BlindIndex.generate_bidx(v, **bi) }
-                else
-                  BlindIndex.generate_bidx(value, **bi)
-                end
-            end
-          end
-        end
-        new_hash
-      end
-
       # memoize for performance
       def has_blind_indexes?
         unless defined?(@has_blind_indexes)
@@ -29,54 +10,38 @@ module BlindIndex
       end
     end
 
-    # ActiveRecord 4.2
     module PredicateBuilder
-      def resolve_column_aliases(klass, hash)
-        new_hash = super
-        if has_blind_indexes?(klass)
-          hash.each do |key, _|
-            if key.respond_to?(:to_sym) && (bi = klass.blind_indexes[key.to_sym]) && !new_hash[key].is_a?(ActiveRecord::StatementCache::Substitute)
-              value = new_hash.delete(key)
-              new_hash[bi[:bidx_attribute]] =
-                if value.is_a?(Array)
-                  value.map { |v| BlindIndex.generate_bidx(v, bi) }
-                else
-                  BlindIndex.generate_bidx(value, bi)
-                end
+      # https://github.com/rails/rails/commit/56f30962b84fc53b76001301fb830c1594fd377e
+      def build(attribute, value, *args)
+        if table.has_blind_indexes? && (bi = table.send(:klass).blind_indexes[attribute.name.to_sym]) && !value.is_a?(ActiveRecord::StatementCache::Substitute)
+          attribute = attribute.relation[bi[:bidx_attribute]]
+          value =
+            if value.is_a?(Array)
+              value.map { |v| BlindIndex.generate_bidx(v, **bi) }
+            else
+              BlindIndex.generate_bidx(value, **bi)
             end
-          end
         end
-        new_hash
-      end
 
-      @@blind_index_cache = {}
-
-      # memoize for performance
-      def has_blind_indexes?(klass)
-        if @@blind_index_cache[klass].nil?
-          @@blind_index_cache[klass] = klass.respond_to?(:blind_indexes)
-        end
-        @@blind_index_cache[klass]
+        super(attribute, value, *args)
       end
     end
 
     module UniquenessValidator
-      if ActiveRecord::VERSION::STRING >= "5.2"
-        def build_relation(klass, attribute, value)
-          if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
-            value = BlindIndex.generate_bidx(value, **bi)
-            attribute = bi[:bidx_attribute]
-          end
-          super(klass, attribute, value)
+      def validate_each(record, attribute, value)
+        klass = record.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
+          value = record.read_attribute_for_validation(bi[:bidx_attribute])
         end
-      else
-        def build_relation(klass, table, attribute, value)
-          if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
-            value = BlindIndex.generate_bidx(value, **bi)
-            attribute = bi[:bidx_attribute]
-          end
-          super(klass, table, attribute, value)
+        super(record, attribute, value)
+      end
+
+      # change attribute name here instead of validate_each for better error message
+      def build_relation(klass, attribute, value)
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
+          attribute = bi[:bidx_attribute]
         end
+        super(klass, attribute, value)
       end
     end
 

data/lib/blind_index/key_generator.rb

@@ -11,7 +11,7 @@ module BlindIndex
       raise ArgumentError, "Missing field for key generation" if bidx_attribute.to_s.empty?
 
       c = "\x7E"*32
-      root_key = hkdf(BlindIndex.decode_key(@master_key), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
+      root_key = hkdf(BlindIndex.decode_key(@master_key, name: "Master key"), salt: table.to_s, info: "#{c}#{bidx_attribute}", length: 32, hash: "sha384")
       hash_hmac("sha256", pack([table, bidx_attribute, bidx_attribute]), root_key)
     end
 

data/lib/blind_index/model.rb

@@ -10,7 +10,7 @@ module BlindIndex
         # check here so we validate rotate options as well
         unknown_keywords = options.keys - [:algorithm, :attribute, :bidx_attribute,
           :callback, :cost, :encode, :expression, :insecure_key, :iterations, :key,
-          :legacy, :master_key, :size, :slow]
+          :key_attribute, :key_table, :legacy, :master_key, :size, :slow, :version]
         raise ArgumentError, "unknown keywords: #{unknown_keywords.join(", ")}" if unknown_keywords.any?
 
         attribute = options[:attribute] || name
@@ -33,9 +33,17 @@ module BlindIndex
         class_method_name = :"generate_#{name}_bidx"
 
         key = options[:key]
-        key ||= -> { BlindIndex.index_key(table: try(:table_name) || collection_name.to_s, bidx_attribute: bidx_attribute, master_key: options[:master_key], encode: false) }
+        key ||= -> { BlindIndex.index_key(table: options[:key_table] || try(:table_name) || collection_name.to_s, bidx_attribute: options[:key_attribute] || bidx_attribute, master_key: options[:master_key], encode: false) }
 
         class_eval do
+          activerecord = defined?(ActiveRecord) && self < ActiveRecord::Base
+
+          if activerecord && ActiveRecord::VERSION::MAJOR >= 6
+            # blind index value isn't really sensitive
+            # but don't need to show it in the Rails console
+            self.filter_attributes += [/\A#{Regexp.escape(bidx_attribute)}\z/]
+          end
+
           @blind_indexes ||= {}
 
           unless respond_to?(:blind_indexes)
@@ -64,25 +72,28 @@ module BlindIndex
             BlindIndex.generate_bidx(value, **blind_indexes[name])
           end
 
-          define_singleton_method method_name do |value|
-            ActiveSupport::Deprecation.warn("Use #{class_method_name} instead")
-            send(class_method_name, value)
-          end
-
           define_method method_name do
-            self.send("#{bidx_attribute}=", self.class.send(class_method_name, send(attribute)))
+            send("#{bidx_attribute}=", self.class.send(class_method_name, send(attribute)))
           end
 
           if callback
-            if defined?(ActiveRecord) && self < ActiveRecord::Base
-              # Active Record
-              # prevent deprecation warnings
-              before_validation method_name, if: -> { changes.key?(attribute.to_s) }
-            else
-              # Mongoid
-              # Lockbox only supports attribute_changed?
-              before_validation method_name, if: -> { send("#{attribute}_changed?") }
+            # TODO reuse module
+            m = Module.new do
+              define_method "#{attribute}=" do |value|
+                result = super(value)
+                send(method_name)
+                result
+              end
+
+              unless activerecord
+                define_method "reset_#{attribute}!" do
+                  result = super()
+                  send(method_name)
+                  result
+                end
+              end
             end
+            prepend m
           end
 
           # use include so user can override

data/lib/blind_index/mongoid.rb

@@ -26,9 +26,9 @@ module BlindIndex
 
               criterion[bidx_key] =
                 if value.is_a?(Array)
-                  value.map { |v| BlindIndex.generate_bidx(v, bi) }
+                  value.map { |v| BlindIndex.generate_bidx(v, **bi) }
                 else
-                  BlindIndex.generate_bidx(value, bi)
+                  BlindIndex.generate_bidx(value, **bi)
                 end
             end
           end
@@ -39,9 +39,18 @@ module BlindIndex
     end
 
     module UniquenessValidator
+      def validate_each(record, attribute, value)
+        klass = record.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
+          value = record.read_attribute_for_validation(bi[:bidx_attribute])
+        end
+        super(record, attribute, value)
+      end
+
+      # change attribute name here instead of validate_each for better error message
       def create_criteria(base, document, attribute, value)
-        if base.respond_to?(:blind_indexes) && (bi = base.blind_indexes[attribute])
-          value = BlindIndex.generate_bidx(value, bi)
+        klass = document.class
+        if klass.respond_to?(:blind_indexes) && (bi = klass.blind_indexes[attribute])
           attribute = bi[:bidx_attribute]
         end
         super(base, document, attribute, value)

data/lib/blind_index/version.rb

@@ -1,3 +1,3 @@
 module BlindIndex
-  VERSION = "1.0.2"
+  VERSION = "2.5.0"
 end

data/lib/blind_index.rb

@@ -1,12 +1,15 @@
 # dependencies
 require "active_support"
+require "argon2/kdf"
+
+# stdlib
 require "openssl"
-require "argon2"
 
 # modules
-require "blind_index/key_generator"
-require "blind_index/model"
-require "blind_index/version"
+require_relative "blind_index/backfill"
+require_relative "blind_index/key_generator"
+require_relative "blind_index/model"
+require_relative "blind_index/version"
 
 module BlindIndex
   class Error < StandardError; end
@@ -18,7 +21,7 @@ module BlindIndex
   self.default_options = {}
 
   def self.master_key
-    @master_key ||= ENV["BLIND_INDEX_MASTER_KEY"]
+    @master_key ||= ENV["BLIND_INDEX_MASTER_KEY"] || (defined?(Lockbox.master_key) && Lockbox.master_key)
   end
 
   def self.generate_bidx(value, key:, **options)
@@ -64,7 +67,7 @@ module BlindIndex
           # use same bounds as rbnacl
           raise BlindIndex::Error, "m must be between 3 and 22" if m < 3 || m > 22
 
-          [Argon2::Engine.hash_argon2id(value, key, t, m, size)].pack("H*")
+          Argon2::KDF.argon2id(value, salt: key, t: t, m: m, p: 1, length: size)
         when :pbkdf2_sha256
           iterations = cost_options[:iterations] || options[:iterations] || (options[:slow] ? 100000 : 10000)
           OpenSSL::PKCS5.pbkdf2_hmac(value, key, iterations, size, "sha256")
@@ -78,7 +81,7 @@ module BlindIndex
           # use same bounds as rbnacl
           raise BlindIndex::Error, "m must be between 3 and 22" if m < 3 || m > 22
 
-          [Argon2::Engine.hash_argon2i(value, key, t, m, size)].pack("H*")
+          Argon2::KDF.argon2i(value, salt: key, t: t, m: m, p: 1, length: size)
         when :scrypt
           n = cost_options[:n] || 4096
           r = cost_options[:r] || 8
@@ -116,42 +119,36 @@ module BlindIndex
     key
   end
 
-  def self.decode_key(key)
+  def self.decode_key(key, name: "Key")
     # decode hex key
     if key.encoding != Encoding::BINARY && key =~ /\A[0-9a-f]{64}\z/i
       key = [key].pack("H*")
     end
 
-    raise BlindIndex::Error, "Key must use binary encoding" if key.encoding != Encoding::BINARY
-    raise BlindIndex::Error, "Key must be 32 bytes" if key.bytesize != 32
+    raise BlindIndex::Error, "#{name} must be 32 bytes (64 hex digits)" if key.bytesize != 32
+    raise BlindIndex::Error, "#{name} must use binary encoding" if key.encoding != Encoding::BINARY
 
     key
   end
+
+  def self.backfill(relation, columns: nil, batch_size: 1000)
+    Backfill.new(relation, columns: columns, batch_size: batch_size).perform
+  end
 end
 
 ActiveSupport.on_load(:active_record) do
-  require "blind_index/extensions"
+  require_relative "blind_index/extensions"
   extend BlindIndex::Model
 
-  if defined?(ActiveRecord::TableMetadata)
-    ActiveRecord::TableMetadata.prepend(BlindIndex::Extensions::TableMetadata)
-  else
-    ActiveRecord::PredicateBuilder.singleton_class.prepend(BlindIndex::Extensions::PredicateBuilder)
-  end
-
+  ActiveRecord::TableMetadata.prepend(BlindIndex::Extensions::TableMetadata)
   ActiveRecord::DynamicMatchers::Method.prepend(BlindIndex::Extensions::DynamicMatchers)
-
-  unless ActiveRecord::VERSION::STRING.start_with?("5.1.")
-    ActiveRecord::Validations::UniquenessValidator.prepend(BlindIndex::Extensions::UniquenessValidator)
-  end
+  ActiveRecord::Validations::UniquenessValidator.prepend(BlindIndex::Extensions::UniquenessValidator)
+  ActiveRecord::PredicateBuilder.prepend(BlindIndex::Extensions::PredicateBuilder)
 end
 
-if defined?(Mongoid)
-  # TODO find better ActiveModel hook
-  require "active_model/callbacks"
-  ActiveModel::Callbacks.include(BlindIndex::Model)
-
-  require "blind_index/mongoid"
+ActiveSupport.on_load(:mongoid) do
+  require_relative "blind_index/mongoid"
+  Mongoid::Document::ClassMethods.include(BlindIndex::Model)
   Mongoid::Criteria.prepend(BlindIndex::Mongoid::Criteria)
   Mongoid::Validatable::UniquenessValidator.prepend(BlindIndex::Mongoid::UniquenessValidator)
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: blind_index
 version: !ruby/object:Gem::Version
-  version: 1.0.2
+  version: 2.5.0
 platform: ruby
 authors:
 - Andrew Kane
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-12-27 00:00:00.000000000 Z
+date: 2024-06-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -16,156 +16,30 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '5'
+        version: '6.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '5'
+        version: '6.1'
 - !ruby/object:Gem::Dependency
-  name: argon2
+  name: argon2-kdf
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2'
+        version: 0.1.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '2'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: minitest
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: attr_encrypted
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: activerecord
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: sqlite3
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: scrypt
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: benchmark-ips
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: lockbox
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0.2'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0.2'
-description: 
-email: andrew@chartkick.com
+        version: 0.1.1
+description:
+email: andrew@ankane.org
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -174,6 +48,7 @@ files:
 - LICENSE.txt
 - README.md
 - lib/blind_index.rb
+- lib/blind_index/backfill.rb
 - lib/blind_index/extensions.rb
 - lib/blind_index/key_generator.rb
 - lib/blind_index/model.rb
@@ -183,7 +58,7 @@ homepage: https://github.com/ankane/blind_index
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -191,15 +66,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.4'
+      version: '3.1'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.2
-signing_key: 
+rubygems_version: 3.5.9
+signing_key:
 specification_version: 4
 summary: Securely search encrypted database fields
 test_files: []