active_enquo 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e643c69efa8243d95e44c59a6d49f20c40a23dc3a914da1d79342fa5d3969ca9
-  data.tar.gz: fcb70caad8b2d949fca672198e269e574924cd63748dfa221164ec6a1eb7842d
+  metadata.gz: 9d538e223b9e7485c4b08cf8ac38f2cc6a08f5d3c3b72426167606fdc7a5fdf5
+  data.tar.gz: 33e80addf4beb8bbe8f5328cbfd093cb70264ec2804d148026cbdabbd36cd1b8
 SHA512:
-  metadata.gz: 7ce8dd63cae62013d026ddf7200878975471bf30c157e059e50b269124e886a219ebb526581038bed3fed1de00500416bfdee7740497c1af2b775c90c46dfd53
-  data.tar.gz: 02a1944917d45d1517cd5143dbe56c05c28f8c8a2fffe5840f317ce865a944aa563a04560f3e39773eadb7f2ca5a0f54835bdab2f4765e5aacc456c9a88f88b8
+  metadata.gz: af6a5e20a215234ce3bce1656a7e02e94f536bf1a3c8d674732d7d49cd8ab5265fe4c306ea730cfe60bba707df5436cf9048d9ce8b85f021fc92639835a0424d
+  data.tar.gz: e9f6b9d0f5f5f5fb5b6fdecd8ba7fdcadf5b7be0c363787d78a2a989bb753b506d9d914122041614aad4cb3032803d6f5075ec02019a81457104471decd4fec7

data/README.md

@@ -116,7 +116,7 @@ You can now, without any further ado, use that attribute in your models as you w
 For example, you can insert a new record:
 
 ```ruby
-User.create!([{name: "Clara Bloggs", username: "cbloggs", date_of_birth: Date(1970, 1, 1)}])
+User.create!([{name: "Clara Bloggs", username: "cbloggs", date_of_birth: Date.new(1970, 1, 1)}])
 ```
 
 When you retrieve a record, the value is there for you to read:
@@ -130,18 +130,18 @@ User.where(username: "cbloggs").first.date_of_birth.to_s  # => "1970-01-01"
 
 This is where things get *neat*.
 
-Performing a query on Enquo-encrypted data is done the same way as on unencrypted data, with one exception: you need to wrap the values of the query in `<Model>.enquo` calls, as in the examples below.
+Performing a query on Enquo-encrypted data is done the same way as on unencrypted data.
 
 You can query for records that have the exact value you're looking for:
 
 ```ruby
-User.where(age: User.enquo(:date_of_birth, Date(1970, 1, 1)))
+User.where(date_of_birth: Date(1970, 1, 1))
 ```
 
 Or you can query for users born less than 50 years ago:
 
 ```ruby
-User.where(age: User.enquo(:date, Date.today - 50.years)..)
+User.where(date_of_birth: (Date.today - 50.years))..)
 ```
 
 This doesn't seem so magical, until you take a peek in the database, and realise that *all the data is still encrypted*:
@@ -154,6 +154,12 @@ psql> SELECT date_of_birth FROM users WHERE username='cbloggs';
 ```
 
 
+## Migrating Existing Data to Encrypted Form
+
+This is a topic on which a lot of words can be written.
+For the sake of tidiness, these words are in [a guide of their own](docs/MIGRATION.md).
+
+
 ## Indexing and Ordering
 
 To maintain [security by default](https://enquo.org/about/threat-models#snapshot-security), ActiveEnquo doesn't provide the ability to `ORDER BY` or index columns by default.
@@ -168,6 +174,7 @@ class User < ApplicationRecord
 end
 ```
 
+
 ### Security Considerations
 
 As the name implies, "reduced security operations" require that the security of the data in the column be lower than [Enquo's default security properties](https://enquo.org/about/threat-models#snapshot-security).

data/active_enquo.gemspec

@@ -27,7 +27,7 @@ Gem::Specification.new do |s|
 
 	s.required_ruby_version = ">= 2.7.0"
 
-	s.add_runtime_dependency "enquo-core", "~> 0.6"
+	s.add_runtime_dependency "enquo-core", "~> 0.7"
 	s.add_runtime_dependency "activerecord", ">= 6"
 
 	s.add_development_dependency "bundler"

data/docs/MIGRATION.md

@@ -0,0 +1,107 @@
+If you have data already in your database that you wish to protect using ActiveEnquo, you can migrate the data into an encrypted form.
+
+There are two approaches that can be used:
+
+* [Direct Migration](#data-migration-with-downtime), which is straightforward but involves application downtime; or
+
+* [Live Migration](#live-data-migration), which is more complicated, but can be done while keeping the application available at all times.
+
+
+# Data Migration (with Downtime)
+
+If your application can withstand being down for a period of time, you can use a simple migration process that encrypts the data and modifies the application appropriate in a single pass.
+It relies on their being a period with nothing accessing the column(s) being migrated, which usually means that the entire application (including background workers and periodic tasks) being shut down, before being restarted.
+The total downtime will depend on how long it takes to encrypt and write all the data being migrated, which is mostly a function of the amount of data being stored.
+
+
+## Step 1: Configure the Encrypted Column(s)
+
+Create an ActiveRecord migration which renames the existing column and creates a new `enquo_*` column with the old name.
+For example, if you already had a `date_of_birth` column, your migration would look like this:
+
+```ruby
+class EncryptUsersDateOfBirth < ActiveRecord::Migration[7.0]
+  def change
+    rename_column :users, :date_of_birth, :date_of_birth_plaintext
+    add_column :users, :date_of_birth, :enquo_date
+    User.enquo_encrypt_columns(date_of_birth_plaintext: :date_of_birth)
+    remove_column :users, :date_of_birth_plaintext
+  end
+end
+```
+
+The `Model.encrypt_columns` method loads all the records in the table, and encrypts the value in the plaintext column and writes it to the corresponding encrypted column.
+
+If you want to encrypt several columns in a single model, you can do so in a single migration, by renaming all the columns and adding `enquo_*` type columns, and then providing the mapping of all columns together in a single `Model.encrypt_columns` call.
+This is the recommended approach, as it improves efficiency because the records only have to be loaded and saved once.
+
+If you want to encrypt columns in multiple models in one downtime, just repeat the above steps for each table and model involved.
+
+
+## Step 2: Modify Queries
+
+When providing data to a query on an encrypted column, you need to make a call to `Model.enquo` in order to encrypt the value for querying.
+
+To continue our `date_of_birth` example above, you need to find any queries that reference the `date_of_birth` column, and modify the code to pass the value for the `date_of_birth` column through a call to `User.enquo(:date_of_birth, <value>)`.
+
+For a query that found all users with a date of birth equal to a query parameter, that originally looked like this:
+
+```ruby
+User.where(date_of_birth: params[:dob])
+```
+
+You'd modify it to look like this, instead:
+
+```ruby
+User.where(date_of_birth: User.enquo(:date_of_birth, params[:dob]))
+```
+
+If the value for the query was passed in as a positional parameter, you just encrypt the value the same way, so that a query might look like this:
+
+```ruby
+User.where("date_of_birth > ? OR date_of_birth IS NULL", User.enquo(:date_of_birth, params[:dob]))
+```
+
+
+## Step 3: Deploy
+
+Once the above changes are all made and committed to revision control, it's time to commence the downtime.
+Shutdown all the application servers, background job workers, and anything else that accesses the database, then perform a normal deployment -- running the database migration process before starting the application again.
+
+The migration may take some time to run, if the table is large.
+
+
+## Step 4: Enjoy Fully Encrypted Data
+
+The column(s) you migrated are now fully protected by Enquo's queryable encryption.
+Relax and enjoy your preferred beverage in celebration!
+
+
+# Live Data Migration
+
+Converting the data in a column to be fully encrypted, while avoiding any application downtime, requires making several changes to the application and database schema in alternating sequence.
+This is necessary to ensure that parts of the application stack running both older and newer code versions can work with the database schema in place at all times.
+
+> # WORK IN PROGRESS
+>
+> This section has not been written out in detail.
+> The short version is:
+>
+> 1. Rename the unencrypted column:
+>   1. `create_column :users, :date_of_birth_plaintext, :date`
+>   2. Modify the model to write changes to `date_of_birth` to `date_of_birth_plaintext` as well
+>   3. Deploy
+>   4. Migration to copy all `date_of_birth` values to `date_of_birth_plaintext`
+>   5. Deploy
+>   6. Modify app to read/query from `date_of_birth_plaintext`, add `date_of_birth` to `User.ignored_columns`
+>   7. Deploy
+>   8. `drop_column :users, :date_of_birth`
+> 2. Create the encrypted column:
+>   1. `create_column :users, :date_of_birth, :enquo_date`
+>   2. Modify the model to write changes to `date_of_birth_plaintext` to `date_of_birth` as well, remove `date_of_birth` from `User.ignored_columns`
+>   3. Deploy
+>   4. Migration to encrypt all `date_of_birth_plaintext` values into `date_of_birth`
+>   5. Deploy
+>   6. Modify app to read/encrypted query from `date_of_birth`, add `date_of_birth_plaintext` to `User.ignored_columns`
+>   7. Deploy
+>   8. `drop_column :users, :date_of_birth_plaintext`

data/e2e_tests/.gitignore

@@ -0,0 +1 @@
+/.pgdbenv

data/e2e_tests/001_direct_migration/exercise_model

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+
+require "active_record"
+require "active_enquo"
+require "pg"
+
+require_relative "../init"
+
+def assert_eq(expected, actual)
+  unless expected == actual
+    $stderr.puts "Expected #{expected.inspect} to equal #{actual.inspect}"
+    exit 1
+  end
+end
+
+def value(f, v)
+  if ENV.key?("USING_ENQUO")
+    People.enquo(f, v)
+  else
+    v
+  end
+end
+
+ActiveRecord::Base.establish_connection(adapter: ENV.fetch("DBTYPE"))
+
+assert_eq(["Meyers"], People.where(first_name: value(:first_name, "Seth")).all.map { |p| p.last_name })
+assert_eq(8, People.where(date_of_birth: value(:date_of_birth, "1980-01-01")..).count)

data/e2e_tests/001_direct_migration/migrations/001_create_people_table.rb

@@ -0,0 +1,11 @@
+class CreatePeopleTable < ActiveRecord::Migration[ENV.fetch("AR_VERSION", "7.0").to_f]
+	def change
+		create_table :people do |t|
+			t.string :first_name
+			t.string :last_name
+			t.date :date_of_birth
+
+			t.timestamps
+		end
+	end
+end

data/e2e_tests/001_direct_migration/migrations/002_encrypt_people_data.rb

@@ -0,0 +1,25 @@
+class EncryptPeopleData < ActiveRecord::Migration[ENV.fetch("AR_VERSION", "7.0").to_f]
+	def up
+		rename_column :people, :first_name, :first_name_plaintext
+		rename_column :people, :last_name, :last_name_plaintext
+		rename_column :people, :date_of_birth, :date_of_birth_plaintext
+
+		add_column :people, :first_name, :enquo_text
+		add_column :people, :last_name, :enquo_text
+		add_column :people, :date_of_birth, :enquo_date
+
+		People.enquo_encrypt_columns(
+			{
+				first_name_plaintext: :first_name,
+				last_name_plaintext: :last_name,
+				date_of_birth_plaintext: :date_of_birth,
+			},
+			# Smol batch size exercises the batching functionality
+			batch_size: 5
+		)
+
+		remove_column :people, :first_name_plaintext
+		remove_column :people, :last_name_plaintext
+		remove_column :people, :date_of_birth_plaintext
+	end
+end

data/e2e_tests/001_direct_migration/run

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+. ../helper.sh
+
+export DBTYPE="postgresql"
+
+check_enquo_pg_db
+clear_pg_db
+
+ar_db_migrate "001"
+load_people
+
+./exercise_model
+
+ar_db_migrate "002"
+USING_ENQUO="y" ./exercise_model

data/e2e_tests/helper.sh

@@ -0,0 +1,34 @@
+if [ -f "../.pgdbenv" ]; then
+	. ../.pgdbenv
+fi
+
+check_enquo_pg_db() {
+	if [ "$(psql -tAc "select count(extname) FROM pg_catalog.pg_extension WHERE extname='pg_enquo'")" != "1" ]; then
+		echo "Specified PostgreSQL database does not have the pg_enquo extension." >&2
+		echo "Check your PG* env vars for correctness, and install the extension if needed." >&2
+		exit 1
+	fi
+}
+
+clear_pg_db() {
+	for tbl in $(psql -tAc "select relname from pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n on n.oid = c.relnamespace where c.relkind='r' and n.nspname = 'public'"); do
+		psql -c "DROP TABLE $tbl" >/dev/null
+	done
+	for seq in $(psql -tAc "select relname from pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n on n.oid = c.relnamespace where c.relkind='S' and n.nspname = 'public'"); do
+		psql -c "DROP SEQUENCE $seq" >/dev/null
+	done
+}
+
+run_ruby() {
+	ruby -r ../init "$@"
+}
+
+ar_db_migrate() {
+	local target_version="$1"
+
+	run_ruby -e "ActiveRecord::MigrationContext.new(['migrations']).up($target_version)" >/dev/null
+}
+
+load_people() {
+	run_ruby -e "People.create!(JSON.parse(File.read('../people.json')))"
+}

data/e2e_tests/init.rb

@@ -0,0 +1,18 @@
+require "active_record"
+require "active_enquo"
+
+ActiveEnquo.root_key = Enquo::RootKey::Static.new("f91c5017a2d946403cc90a688266ff32d186aa2a00efd34dcaa86be802e179d0")
+
+DBTYPE = ENV.fetch("DBTYPE")
+
+case DBTYPE
+when "postgresql"
+	require "pg"
+else
+	raise "Unsupported DBTYPE: #{DBTYPE.inspect}"
+end
+
+class People < ActiveRecord::Base
+end
+
+ActiveRecord::Base.establish_connection(adapter: DBTYPE)

data/e2e_tests/people.json

@@ -0,0 +1 @@
+[{"first_name":"Sergio","last_name":"Leone","date_of_birth":"1929-01-03"},{"first_name":"Bradley","last_name":"Cooper","date_of_birth":"1975-01-05"},{"first_name":"Robert","last_name":"Duvall","date_of_birth":"1931-01-05"},{"first_name":"Kate","last_name":"Moss","date_of_birth":"1974-01-16"},{"first_name":"Laura","last_name":"Schlessinger","date_of_birth":"1947-01-16"},{"first_name":"Jason","last_name":"Segel","date_of_birth":"1980-01-18"},{"first_name":"Pete","last_name":"Buttigieg","date_of_birth":"1982-01-19"},{"first_name":"Mischa","last_name":"Barton","date_of_birth":"1986-01-24"},{"first_name":"Shirley","last_name":"Mason","date_of_birth":"1923-01-25"},{"first_name":"Boris","last_name":"Yeltsin","date_of_birth":"1931-02-01"},{"first_name":"Tom","last_name":"Wilkinson","date_of_birth":"1948-02-05"},{"first_name":"Tony","last_name":"Iommi","date_of_birth":"1948-02-19"},{"first_name":"Sri","last_name":"Srinivasan","date_of_birth":"1967-02-23"},{"first_name":"Tony","last_name":"Randall","date_of_birth":"1920-02-26"},{"first_name":"David","last_name":"Blaine","date_of_birth":"1973-04-04"},{"first_name":"Muddy","last_name":"Waters","date_of_birth":"1915-04-04"},{"first_name":"Danny","last_name":"Almonte","date_of_birth":"1987-04-07"},{"first_name":"Meghann","last_name":"Shaughnessy","date_of_birth":"1979-04-13"},{"first_name":"Maisie","last_name":"Williams","date_of_birth":"1997-04-15"},{"first_name":"Immanuel","last_name":"Kant","date_of_birth":"1724-04-22"},{"first_name":"John","last_name":"Oliver","date_of_birth":"1977-04-23"},{"first_name":"Penelope","last_name":"Cruz","date_of_birth":"1974-04-28"},{"first_name":"Anne","last_name":"Parillaud","date_of_birth":"1961-05-06"},{"first_name":"Cate","last_name":"Blanchett","date_of_birth":"1969-05-14"},{"first_name":"Tucker","last_name":"Carlson","date_of_birth":"1969-05-16"},{"first_name":"Nancy","last_name":"Kwan","date_of_birth":"1939-05-19"},{"first_name":"Melissa","last_name":"Etheridge","date_of_birth":"1961-05-29"},{"first_name":"Marissa","last_name":"Mayer","date_of_birth":"1975-05-30"},{"first_name":"Marvin","last_name":"Hamlisch","date_of_birth":"1944-06-02"},{"first_name":"Gwendolyn","last_name":"Brooks","date_of_birth":"1917-06-07"},{"first_name":"Paul","last_name":"Lynde","date_of_birth":"1927-06-13"},{"first_name":"Paul","last_name":"McCartney","date_of_birth":"1942-06-18"},{"first_name":"Susan","last_name":"Hayward","date_of_birth":"1917-06-30"},{"first_name":"Léa","last_name":"Seydoux","date_of_birth":"1985-07-01"},{"first_name":"Amanda","last_name":"Knox","date_of_birth":"1987-07-09"},{"first_name":"Cindy","last_name":"Sheehan","date_of_birth":"1957-07-10"},{"first_name":"Bill","last_name":"Cosby","date_of_birth":"1937-07-12"},{"first_name":"Raymond","last_name":"Chandler","date_of_birth":"1888-07-23"},{"first_name":"Christopher","last_name":"Nolan","date_of_birth":"1970-07-30"},{"first_name":"Melanie","last_name":"Griffith","date_of_birth":"1957-08-09"},{"first_name":"Freddie","last_name":"Gray","date_of_birth":"1989-08-16"},{"first_name":"Steve","last_name":"Case","date_of_birth":"1958-08-21"},{"first_name":"Cal","last_name":"Ripken","date_of_birth":"1960-08-24"},{"first_name":"Regis","last_name":"Philbin","date_of_birth":"1931-08-25"},{"first_name":"Shania","last_name":"Twain","date_of_birth":"1965-08-28"},{"first_name":"Adam","last_name":"Curry","date_of_birth":"1964-09-03"},{"first_name":"Rachel","last_name":"Hunter","date_of_birth":"1969-09-09"},{"first_name":"Bashar","last_name":"al-Assad","date_of_birth":"1965-09-11"},{"first_name":"Amy","last_name":"Poehler","date_of_birth":"1971-09-16"},{"first_name":"Jim","last_name":"Thompson","date_of_birth":"1906-09-27"},{"first_name":"Matt","last_name":"Damon","date_of_birth":"1970-10-08"},{"first_name":"Leopold","last_name":"Senghor","date_of_birth":"1906-10-09"},{"first_name":"William","last_name":"Penn","date_of_birth":"1644-10-14"},{"first_name":"Rebecca","last_name":"Romijn","date_of_birth":"1971-11-06"},{"first_name":"Auguste","last_name":"Rodin","date_of_birth":"1840-11-12"},{"first_name":"Nate","last_name":"Parker","date_of_birth":"1979-11-18"},{"first_name":"Larry","last_name":"Bird","date_of_birth":"1956-12-07"},{"first_name":"Bobby","last_name":"Flay","date_of_birth":"1964-12-10"},{"first_name":"Chris","last_name":"Evert","date_of_birth":"1954-12-21"},{"first_name":"Frank","last_name":"Zappa","date_of_birth":"1940-12-21"},{"first_name":"Estella","last_name":"Warren","date_of_birth":"1978-12-23"},{"first_name":"Carlos","last_name":"Castaneda","date_of_birth":"1925-12-25"},{"first_name":"Seth","last_name":"Meyers","date_of_birth":"1973-12-28"}]

data/e2e_tests/run

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+for t in [0-9]*; do
+	rv="0"
+	cd "$t"
+
+	./run || rv="$?"
+
+	if [ "$rv" != "0" ]; then
+		echo "Test $i failed with exit code $rv" >&2
+		exit 1
+	fi
+done
+
+echo "All tests passed."

data/lib/active_enquo.rb

@@ -21,7 +21,57 @@ module ActiveEnquo
 	RootKey = Enquo::RootKey
 
 	module ActiveRecord
-		module ModelExtension
+		module QueryFilterMangler
+			class Ciphertext < String
+			end
+			private_constant :Ciphertext
+
+			private
+
+			def mangle_query_filter(a)
+				args = a.first
+				if args.is_a?(Hash)
+					args.keys.each do |attr|
+						next unless enquo_attr?(attr)
+
+						if args[attr].is_a?(Array)
+							args[attr] = args[attr].map { |v| maybe_enquo(attr, v) }
+						elsif args[attr].is_a?(Range)
+							r = args[attr]
+							args[attr] = if r.exclude_end?
+								if r.begin.nil?
+									...maybe_enquo(attr, r.end)
+								elsif r.end.nil?
+									(maybe_enquo(attr.r.begin)...)
+								else
+									maybe_enquo(attr.r.begin)...maybe_enquo(attr, r.end)
+								end
+							else
+								if r.begin.nil?
+									..maybe_enquo(attr, r.end)
+								elsif r.end.nil?
+									maybe_enquo(attr.r.begin)..
+								else
+									maybe_enquo(attr.r.begin)..maybe_enquo(attr, r.end)
+								end
+							end
+						else
+							args[attr] = maybe_enquo(attr, args[attr])
+						end
+					end
+				end
+			end
+
+			def maybe_enquo(attr, v)
+				if v.nil? || v.is_a?(Ciphertext) || v.is_a?(::ActiveRecord::StatementCache::Substitute)
+					v
+				else
+					Ciphertext.new(enquo(attr, v))
+				end
+			end
+		end
+
+		module BaseExtension
 			extend ActiveSupport::Concern
 
 			def _read_attribute(attr_name, &block)
@@ -51,10 +101,7 @@ module ActiveEnquo
 					relation = self.class.arel_table.name
 					field = ::ActiveEnquo.root.field(relation, attr_name)
 					attr_opts = self.class.enquo_attribute_options.fetch(attr_name.to_sym, {})
-					safety = if attr_opts[:enable_reduced_security_operations]
-						:unsafe
-					end
-					db_value = t.encrypt(value, @attributes.fetch_value(@primary_key).to_s, field, safety: safety, no_query: attr_opts[:no_query])
+					db_value = t.encrypt(value, @attributes.fetch_value(@primary_key).to_s, field, **attr_opts)
 					@attributes.write_from_user(attr_name, db_value)
 				else
 					super
@@ -62,28 +109,114 @@ module ActiveEnquo
 			end
 
 			module ClassMethods
-				def enquo(attr_name, value)
+				include QueryFilterMangler
+
+				def find_by(*a)
+					mangle_query_filter(a)
+					super
+				end
+
+				def enquo(attr_name, value_or_meta_id, maybe_value = nil)
+					meta_id, value = if value_or_meta_id.is_a?(Symbol)
+						[value_or_meta_id, maybe_value]
+					else
+						[nil, value_or_meta_id]
+					end
+
 					t = self.attribute_types[attr_name.to_s]
 					if t.is_a?(::ActiveEnquo::Type)
 						relation = self.arel_table.name
 						field = ::ActiveEnquo.root.field(relation, attr_name)
-						t.encrypt(value, "", field, safety: :unsafe)
+						if meta_id.nil?
+							t.encrypt(value, "", field, enable_reduced_security_operations: true)
+						else
+							t.encrypt_metadata_value(meta_id, value, field)
+						end
 					else
 						raise ArgumentError, "Cannot produce encrypted value on a non-enquo attribute '#{attr_name}'"
 					end
 				end
 
+				def unenquo(attr_name, value, ctx)
+					t = self.attribute_types[attr_name.to_s]
+					if t.is_a?(::ActiveEnquo::Type)
+						relation = self.arel_table.name
+						field = ::ActiveEnquo.root.field(relation, attr_name)
+						begin
+							t.decrypt(value, ctx, field)
+						rescue Enquo::Error
+							t.decrypt(value, "", field)
+						end
+					else
+						raise ArgumentError, "Cannot decrypt value on a non-enquo attribute '#{attr_name}'"
+					end
+				end
+
+				def enquo_attr?(attr_name)
+					self.attribute_types[attr_name.to_s].is_a?(::ActiveEnquo::Type)
+				end
+
 				def enquo_attr(attr_name, opts)
+					if opts.key?(:default)
+						default_value = opts.delete(:default)
+						after_initialize do
+							next if persisted?
+							next unless self.send(attr_name).nil?
+							self.send(:"#{attr_name}=", default_value.duplicable? ? default_value.dup : default_value)
+						end
+					end
+
 					enquo_attribute_options[attr_name] = @enquo_attribute_options[attr_name].merge(opts)
 				end
 
 				def enquo_attribute_options
 					@enquo_attribute_options ||= Hash.new({})
 				end
+
+				def enquo_encrypt_columns(column_map, batch_size: 10_000)
+					plaintext_columns = column_map.keys
+					relation = self.arel_table.name
+					in_progress = true
+					self.reset_column_information
+
+					while in_progress
+						self.transaction do
+							# The .where("0=1") here is a dummy condition so that the q.or in the .each will work properly
+							q = self.select(self.primary_key).select(plaintext_columns).where("0=1")
+							column_map.each do |pt_col, ct_col|
+								q = q.or(self.where(ct_col => nil).where.not(pt_col => nil))
+							end
+
+							q = q.limit(batch_size).lock
+
+							rows = ::ActiveRecord::Base.connection.exec_query(q.to_sql)
+							if rows.length == 0
+								in_progress = false
+							else
+								rows.each do |row|
+									values = Hash[column_map.map do |pt_col, ct_col|
+										field = ::ActiveEnquo.root.field(relation, ct_col)
+										attr_opts = self.enquo_attribute_options.fetch(ct_col.to_sym, {})
+										t = self.attribute_types[ct_col.to_s]
+										db_value = t.encrypt(row[pt_col.to_s], row[self.primary_key].to_s, field, **attr_opts)
+
+										[ct_col, db_value]
+									end]
+
+									self.where(self.primary_key => row[self.primary_key]).update_all(values)
+								end
+							end
+						end
+					end
+				end
 			end
 		end
 
 		module TableDefinitionExtension
+			def enquo_boolean(name, **options)
+				column(name, :enquo_boolean, **options)
+			end
+
 			def enquo_bigint(name, **options)
 				column(name, :enquo_bigint, **options)
 			end
@@ -96,11 +229,28 @@ module ActiveEnquo
 				column(name, :enquo_text, **options)
 			end
 		end
+
+		module RelationExtension
+			include QueryFilterMangler
+			extend ActiveSupport::Concern
+
+			def where(*a)
+				mangle_query_filter(a)
+				super
+			end
+
+			def exists?(*a)
+				mangle_query_filter(a)
+				super
+			end
+
+		end
 	end
 
 	module Postgres
 		module ConnectionAdapter
 			def initialize_type_map(m = type_map)
+				m.register_type "enquo_boolean", ActiveEnquo::Type::Boolean.new
 				m.register_type "enquo_bigint", ActiveEnquo::Type::Bigint.new
 				m.register_type "enquo_date", ActiveEnquo::Type::Date.new
 				m.register_type "enquo_text", ActiveEnquo::Type::Text.new
@@ -111,13 +261,35 @@ module ActiveEnquo
 	end
 
 	class Type < ::ActiveRecord::Type::Value
+		class Boolean < Type
+			def type
+				:enquo_boolean
+			end
+
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_boolean(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
+			end
+
+			def decrypt(value, context, field)
+				field.decrypt_boolean(value, context)
+			end
+		end
+
 		class Bigint < Type
 			def type
 				:enquo_bigint
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
-				field.encrypt_i64(value, context, safety: safety, no_query: no_query)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_i64(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
 			end
 
 			def decrypt(value, context, field)
@@ -130,9 +302,13 @@ module ActiveEnquo
 				:enquo_date
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
 				value = cast_to_date(value)
-				field.encrypt_date(value, context, safety: safety, no_query: no_query)
+				if value.nil?
+					value
+				else
+					field.encrypt_date(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
 			end
 
 			def decrypt(value, context, field)
@@ -146,6 +322,8 @@ module ActiveEnquo
 					value
 				elsif value.respond_to?(:to_date)
 					value.to_date
+				elsif value.nil? || (value.respond_to?(:empty?) && value.empty?) || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					nil
 				else
 					Time.parse(value.to_s).to_date
 				end
@@ -157,12 +335,33 @@ module ActiveEnquo
 				:enquo_text
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
-				field.encrypt_text(value, context, safety: safety, no_query: no_query)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false, enable_ordering: false)
+				if enable_ordering && !enable_reduced_security_operations
+					raise ArgumentError, "Cannot enable ordering on an Enquo attribute unless Reduced Security Operations are enabled"
+				end
+
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_text(value.respond_to?(:encode) ? value.encode("UTF-8") : value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query, order_prefix_length: enable_ordering ? 8 : nil)
+				end
+			end
+
+			def encrypt_metadata_value(name, value, field)
+				case name
+				when :length
+					field.encrypt_text_length_query(value)
+				else
+					raise ArgumentError, "Unknown metadata name for Text field: #{name.inspect}"
+				end
 			end
 
 			def decrypt(value, context, field)
-				field.decrypt_text(value, context)
+				if value.nil?
+					nil
+				else
+					field.decrypt_text(value, context)
+				end
 			end
 		end
 	end
@@ -171,11 +370,21 @@ module ActiveEnquo
 		class Initializer < Rails::Railtie
 			initializer "active_enquo.root_key" do |app|
 				if app
-					if root_key = app.credentials.active_enquo.root_key
-						ActiveEnquo.root_key = Enquo::RootKey::Static.new(root_key)
+					if app.credentials
+						if app.credentials.active_enquo
+							if root_key = app.credentials.active_enquo.root_key
+								ActiveEnquo.root_key = Enquo::RootKey::Static.new(root_key)
+							else
+								Rails.logger.warn "Could not initialize ActiveEnquo, as no active_enquo.root_key credential was found for this environment"
+							end
+						else
+							Rails.logger.warn "Could not initialize ActiveEnquo, as no active_enquo credentials were found for this environment"
+						end
 					else
-						Rails.warn "Could not initialize ActiveEnquo, as no active_enquo.root_key credential was found for this environment"
+						Rails.logger.warn "Could not initialize ActiveEnquo, as no credentials were found for this environment"
 					end
+				else
+					Rails.logger.warn "Could not initialize ActiveEnquo, as no app was found for this environment"
 				end
 			end
 		end
@@ -183,13 +392,15 @@ module ActiveEnquo
 end
 
 ActiveSupport.on_load(:active_record) do
-	::ActiveRecord::Base.send :include, ActiveEnquo::ActiveRecord::ModelExtension
+	::ActiveRecord::Relation.prepend ActiveEnquo::ActiveRecord::RelationExtension
+	::ActiveRecord::Base.include ActiveEnquo::ActiveRecord::BaseExtension
 
 	::ActiveRecord::ConnectionAdapters::Table.include ActiveEnquo::ActiveRecord::TableDefinitionExtension
 	::ActiveRecord::ConnectionAdapters::TableDefinition.include ActiveEnquo::ActiveRecord::TableDefinitionExtension
 
 	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend ActiveEnquo::Postgres::ConnectionAdapter
 
+	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_boolean] = { name: "enquo_boolean" }
 	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_bigint] = { name: "enquo_bigint" }
 	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_date]   = { name: "enquo_date" }
 	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_text]   = { name: "enquo_text" }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: active_enquo
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Matt Palmer
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-11-08 00:00:00.000000000 Z
+date: 2023-04-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: enquo-core
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0.7'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0.7'
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -195,6 +195,16 @@ files:
 - README.md
 - active_enquo.gemspec
 - docs/DEVELOPMENT.md
+- docs/MIGRATION.md
+- e2e_tests/.gitignore
+- e2e_tests/001_direct_migration/exercise_model
+- e2e_tests/001_direct_migration/migrations/001_create_people_table.rb
+- e2e_tests/001_direct_migration/migrations/002_encrypt_people_data.rb
+- e2e_tests/001_direct_migration/run
+- e2e_tests/helper.sh
+- e2e_tests/init.rb
+- e2e_tests/people.json
+- e2e_tests/run
 - lib/active_enquo.rb
 homepage: https://enquo.org
 licenses: []