active_enquo 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9c0103ba2927e84bdc1c6fda73e7103ffe090316fac200d7d429a95e4f4632f0
-  data.tar.gz: 11d5f5d66a753fe18f79f198c7b243d9fa3b194929c6e2e63c6f4a93ce25ed06
+  metadata.gz: 9d538e223b9e7485c4b08cf8ac38f2cc6a08f5d3c3b72426167606fdc7a5fdf5
+  data.tar.gz: 33e80addf4beb8bbe8f5328cbfd093cb70264ec2804d148026cbdabbd36cd1b8
 SHA512:
-  metadata.gz: 693f14e986cca46c810505664833cd4e211c170cd287d43fbbb429ba7a5ef0e830404d71597a314e97cdde4fe911edf14066f9af446510ce82c05b3e73507c6d
-  data.tar.gz: afba7f08679b1b45f6484f9a3bf1b870fc2a824edec9b8b33483f40b7661c940f55d8135992885011d25a4c57d6c02ecb5f069ebde0bed0d59daec85d6ad301d
+  metadata.gz: af6a5e20a215234ce3bce1656a7e02e94f536bf1a3c8d674732d7d49cd8ab5265fe4c306ea730cfe60bba707df5436cf9048d9ce8b85f021fc92639835a0424d
+  data.tar.gz: e9f6b9d0f5f5f5fb5b6fdecd8ba7fdcadf5b7be0c363787d78a2a989bb753b506d9d914122041614aad4cb3032803d6f5075ec02019a81457104471decd4fec7

data/README.md

@@ -3,12 +3,12 @@ This allows you to keep the data you store safe, by encrypting it, without compr
 
 Sounds like magic?
 Well, maybe a little bit.
-Read our [how it works](https://enquo.org/how-it-works) if you're interested in the details, or read on for how to use it.
+Read our [how it works](https://enquo.org/how-it-works) if you're interested in the gory cryptographic details, or read on for how to use it.
 
 
 # Pre-requisites
 
-In order to make use of this extension, you must be running Postgres 10 or higher, with the [`pg_enquo`](https://github.com/enquo/pg_enquo) extension enabled in the database you're working in.
+In order to make use of ActiveRecord extension, you must be running Postgres 11 or higher, with the [`pg_enquo`](https://github.com/enquo/pg_enquo) extension enabled in the database you're working in.
 See [the `pg_enquo` installation guide](https://github.com/enquo/pg_enquo/tree/main/doc/installation.md) for instructions on how to install `pg_enquo`.
 
 Also, if you're installing this gem from source, you'll need a reasonably recent [Rust](https://rust-lang.org) toolchain installed.
@@ -16,33 +16,41 @@ Also, if you're installing this gem from source, you'll need a reasonably recent
 
 # Installation
 
-It's a gem:
+It's a gem, so the usual methods should work Just Fine:
 
-    gem install active_enquo
-
-There's also the wonders of [the Gemfile](http://bundler.io):
+```sh
+gem install active_enquo
+# OR
+echo "gem 'active_enquo'" >> Gemfile
+```
 
-    gem 'active_enquo'
+On macOS, and Linux `x86-64`/`aarch64`, you'll get a pre-built binary gem that contains everything you need.
+For other platforms, you'll need to have Rust 1.59.0 or later installed in order to build the native code portion of the gem.
 
-If you're the sturdy type that likes to run from git:
 
-    rake install
+# Configuration
 
-Or, if you've eschewed the convenience of Rubygems entirely, then you
-presumably know what to do already.
+The only setting that ActiveEnquo needs is to be given a "root" key, which is used to derive the keys which are used to actually encrypt data.
 
 
-# Configuration
+## Step 1: Generate a Root Key
 
-The only setting that ActiveEnquo needs is to be given a "root" key, which is used to derive the keys which are used to actually encrypt data.
-This key ***MUST*** be generated by a cryptographically-secure random number generator, and must also be 64 hex digits in length.
+The ActiveEnquo root key ***MUST*** be generated by a cryptographically-secure random number generator, and must also be 64 hex digits in length.
 A good way to generate this key is with the `SecureRandom` module:
 
 ```sh
 ruby -r securerandom -e 'puts SecureRandom.hex(32)'
 ```
 
-With this key in hand, you can store it in the Rails credential store, like this:
+
+## Step 2: Configure Your Application
+
+With this key in hand, you need to store it somewhere.
+
+
+### Using Rails Credential Store (Recommended)
+
+The recommended way to store your root key, at present, is in the [Rails credentials store](https://guides.rubyonrails.org/security.html#custom-credentials).
 
 1. Open up the Rails credentials editor:
 
@@ -59,7 +67,11 @@ With this key in hand, you can store it in the Rails credential store, like this
 
 3. Save and exit the editor.  Commit the changes to your revision control system.
 
-This only works if you are using Rails, of course; if you're using ActiveRecord by itself, you must set the root key yourself during application initialization.
+
+### Direct Assignment (Only If You Must)
+
+Using the Rails credential store only works if you are using Rails, of course.
+If you're using ActiveRecord by itself, you must set the root key yourself during application initialization.
 You do this by assigning a `RootKey` to `ActiveEnquo.root_key`, like this:
 
 ```ruby
@@ -67,9 +79,12 @@ You do this by assigning a `RootKey` to `ActiveEnquo.root_key`, like this:
 ActiveEnquo.root_key = ActiveEnquo::RootKey::Static.new("0000000000000000000000000000000000000000000000000000000000000000")
 ```
 
-However, this leaves the key exposed to anyone who comes along and takes a glance at your code.
-Losing control of this root key is catastrophic for the security of your system.
-Instead, you should use a secrets vault of some sort to store the key, and pass it into your initialization code somehow.
+Preferably, you would pass the key into your application via, say, an environment variable, and then immediately clear the environment variable:
+
+```ruby
+ActiveEnquo.root_key = ActiveEnquo::RootKey::Static.new(ENV.fetch("ENQUO_ROOT_KEY"))
+ENV.delete("ENQUO_ROOT_KEY")
+```
 
 Support for cloud keystores, such as AWS KMS, GCP KMS, Azure KeyVault, and HashiCorp Vault, will be implemented sooner or later.
 If you have a burning desire to see that more on the "sooner" end than "later", PRs are welcome.
@@ -77,67 +92,78 @@ If you have a burning desire to see that more on the "sooner" end than "later",
 
 # Usage
 
-Start by creating a column in your database that uses one of the [available enquo types](https://github.com/enquo/pg_enquo/doc/data_types), with a Rails migration:
+We try to make using ActiveEnquo as simple as possible.
+
+
+## Create Your Encrypted Column
+
+Start by creating a column in your database that uses one of the [available `enquo_*` types](https://github.com/enquo/pg_enquo/tree/main/doc/data_types), with a Rails migration:
 
 ```ruby
 class AddEncryptedBigintColumn < ActiveRecord::Migration[6.0]
   def change
-    add_column :users, :age, :enquo_bigint
+    add_column :users, :date_of_birth, :enquo_date
   end
 end
 ```
 
+Apply this migration in the usual fashion (`rails db:migrate`).
+
 
 ## Reading and Writing
 
-You can now use that attribute in your models as you would normally.
-For example, insert a new record:
+You can now, without any further ado, use that attribute in your models as you would normally.
+For example, you can insert a new record:
 
 ```ruby
-User.create!([{name: "Clara Bloggs", username: "cbloggs", age: 42}])
+User.create!([{name: "Clara Bloggs", username: "cbloggs", date_of_birth: Date.new(1970, 1, 1)}])
 ```
 
 When you retrieve a record, the value is there for you to read:
 
 ```ruby
-User.where(username: "cbloggs").first.age  # => 42
+User.where(username: "cbloggs").first.date_of_birth.to_s  # => "1970-01-01"
 ```
 
-So far, nothing more spectacular than what AR Encryption will get you.
-The fun begins now...
-
 
 ## Querying
 
-You can query for records with an exact age:
+This is where things get *neat*.
+
+Performing a query on Enquo-encrypted data is done the same way as on unencrypted data.
+
+You can query for records that have the exact value you're looking for:
 
 ```ruby
-User.where(age: User.enquo(:age, 42))
+User.where(date_of_birth: Date(1970, 1, 1))
 ```
 
-Or you can query for records with an age of less than 50:
+Or you can query for users born less than 50 years ago:
 
 ```ruby
-# This is AR's idiomatic way of saying "less-than"
-User.where(age: User.enquo(:age, ...50))
+User.where(date_of_birth: (Date.today - 50.years))..)
 ```
 
-*However*, if you examine the record in the database, it's encrypted:
+This doesn't seem so magical, until you take a peek in the database, and realise that *all the data is still encrypted*:
 
 ```sh
-psql> SELECT age FROM users WHERE username='cbloggs';
+psql> SELECT date_of_birth FROM users WHERE username='cbloggs';
   age
 -------
- {"ct":[<lots of numbers>],"ore":[<lots and LOTS of numbers>]}
+ {"v1":{"a":[<lots of numbers>],"y":[<lots and LOTS of numbers>],<etc etc>}}
 ```
 
-And that, as they say, is that.
+
+## Migrating Existing Data to Encrypted Form
+
+This is a topic on which a lot of words can be written.
+For the sake of tidiness, these words are in [a guide of their own](docs/MIGRATION.md).
 
 
 ## Indexing and Ordering
 
-To maintain [security](https://enquo.org/about/threat-models#snapshot-security), ActiveEnquo isn't able to `ORDER BY` or index columns.
-This is fine for many situations -- many columns don't need indexes.
+To maintain [security by default](https://enquo.org/about/threat-models#snapshot-security), ActiveEnquo doesn't provide the ability to `ORDER BY` or index columns by default.
+This is fine for many situations -- many columns don't need indexes or to be ordered in a query.
 
 For those columns that *do* need indexes or `ORDER BY` support, you can enable support for them by setting the `enable_reduced_security_operations` flag on the attribute, like this:
 
@@ -148,10 +174,12 @@ class User < ApplicationRecord
 end
 ```
 
-### SECURITY ALERT
+
+### Security Considerations
 
 As the name implies, "reduced security operations" require that the security of the data in the column be lower than [Enquo's default security properties](https://enquo.org/about/threat-models#snapshot-security).
-In particular, extra data is stored in the value which can be used by an attacker to:
+Specifically, extra data needs to be stored in the value to enable indexing and ordering.
+This extra data can be used by an attacker to:
 
 * Identify all rows which have the same value for the column (although not what that value actually *is*); and
 
@@ -174,22 +202,14 @@ class User < ApplicationRecord
 end
 ```
 
+More accurate indications of the disk space requirements for the supported data types can be found in [the description of each data type](https://github.com/enquo/pg_enquo/tree/main/doc/data_types).
 
-# Future Developments
 
-These are some of the things that are definitely planned to be added to ActiveEnquo in the nearish future.
-
-* **Support for key rotation**: this isn't tricky, just a bit fiddly.
-  Encrypted values have a "key ID" associated with them, so we can find which values are out-of-date, but multiple keys need to useable for decryption.
-  Closely related to this is **support for renaming columns**, which is problematic because keys are derived based on the column name.
-  Again, key IDs (to find values encrypted with the previous column name) and the ability to attempt decryption with a key based on the previous column name sorts this out.
-
-* **Strings**: a great deal of the sensitive data that needs protecting is in the form of strings.
-  Querying strings is somewhat more involved than numeric data, and so the means of encrypting strings such that they're queryable *and* secure are more complex.
-  Enquo cannot be a really useful library for supporting encrypted querying until at least some common string query operations are supported, though, so it is important that this be implemented.
+# Future Developments
 
-* **Other data types**: while you can go a long way with strings and bigints, part of the power of SQL is the sheer variety of interesting data types it has, and the variety of things you can do with them.
-  So more integer types, floats, decimals, dates, times, timestamps, and more, are all on the cards for future development.
+ActiveEnquo is far from finished.
+Many more features are coming in the future.
+See [the Enquo project roadmap](https://enquo.org/roadmap) for details of what we're still intending to implement.
 
 
 # Contributing

data/active_enquo.gemspec

@@ -27,7 +27,7 @@ Gem::Specification.new do |s|
 
 	s.required_ruby_version = ">= 2.7.0"
 
-	s.add_runtime_dependency "enquo-core", "~> 0.6"
+	s.add_runtime_dependency "enquo-core", "~> 0.7"
 	s.add_runtime_dependency "activerecord", ">= 6"
 
 	s.add_development_dependency "bundler"

data/docs/MIGRATION.md

@@ -0,0 +1,107 @@
+If you have data already in your database that you wish to protect using ActiveEnquo, you can migrate the data into an encrypted form.
+
+There are two approaches that can be used:
+
+* [Direct Migration](#data-migration-with-downtime), which is straightforward but involves application downtime; or
+
+* [Live Migration](#live-data-migration), which is more complicated, but can be done while keeping the application available at all times.
+
+
+# Data Migration (with Downtime)
+
+If your application can withstand being down for a period of time, you can use a simple migration process that encrypts the data and modifies the application appropriate in a single pass.
+It relies on their being a period with nothing accessing the column(s) being migrated, which usually means that the entire application (including background workers and periodic tasks) being shut down, before being restarted.
+The total downtime will depend on how long it takes to encrypt and write all the data being migrated, which is mostly a function of the amount of data being stored.
+
+
+## Step 1: Configure the Encrypted Column(s)
+
+Create an ActiveRecord migration which renames the existing column and creates a new `enquo_*` column with the old name.
+For example, if you already had a `date_of_birth` column, your migration would look like this:
+
+```ruby
+class EncryptUsersDateOfBirth < ActiveRecord::Migration[7.0]
+  def change
+    rename_column :users, :date_of_birth, :date_of_birth_plaintext
+    add_column :users, :date_of_birth, :enquo_date
+    User.enquo_encrypt_columns(date_of_birth_plaintext: :date_of_birth)
+    remove_column :users, :date_of_birth_plaintext
+  end
+end
+```
+
+The `Model.encrypt_columns` method loads all the records in the table, and encrypts the value in the plaintext column and writes it to the corresponding encrypted column.
+
+If you want to encrypt several columns in a single model, you can do so in a single migration, by renaming all the columns and adding `enquo_*` type columns, and then providing the mapping of all columns together in a single `Model.encrypt_columns` call.
+This is the recommended approach, as it improves efficiency because the records only have to be loaded and saved once.
+
+If you want to encrypt columns in multiple models in one downtime, just repeat the above steps for each table and model involved.
+
+
+## Step 2: Modify Queries
+
+When providing data to a query on an encrypted column, you need to make a call to `Model.enquo` in order to encrypt the value for querying.
+
+To continue our `date_of_birth` example above, you need to find any queries that reference the `date_of_birth` column, and modify the code to pass the value for the `date_of_birth` column through a call to `User.enquo(:date_of_birth, <value>)`.
+
+For a query that found all users with a date of birth equal to a query parameter, that originally looked like this:
+
+```ruby
+User.where(date_of_birth: params[:dob])
+```
+
+You'd modify it to look like this, instead:
+
+```ruby
+User.where(date_of_birth: User.enquo(:date_of_birth, params[:dob]))
+```
+
+If the value for the query was passed in as a positional parameter, you just encrypt the value the same way, so that a query might look like this:
+
+```ruby
+User.where("date_of_birth > ? OR date_of_birth IS NULL", User.enquo(:date_of_birth, params[:dob]))
+```
+
+
+## Step 3: Deploy
+
+Once the above changes are all made and committed to revision control, it's time to commence the downtime.
+Shutdown all the application servers, background job workers, and anything else that accesses the database, then perform a normal deployment -- running the database migration process before starting the application again.
+
+The migration may take some time to run, if the table is large.
+
+
+## Step 4: Enjoy Fully Encrypted Data
+
+The column(s) you migrated are now fully protected by Enquo's queryable encryption.
+Relax and enjoy your preferred beverage in celebration!
+
+
+# Live Data Migration
+
+Converting the data in a column to be fully encrypted, while avoiding any application downtime, requires making several changes to the application and database schema in alternating sequence.
+This is necessary to ensure that parts of the application stack running both older and newer code versions can work with the database schema in place at all times.
+
+> # WORK IN PROGRESS
+>
+> This section has not been written out in detail.
+> The short version is:
+>
+> 1. Rename the unencrypted column:
+>   1. `create_column :users, :date_of_birth_plaintext, :date`
+>   2. Modify the model to write changes to `date_of_birth` to `date_of_birth_plaintext` as well
+>   3. Deploy
+>   4. Migration to copy all `date_of_birth` values to `date_of_birth_plaintext`
+>   5. Deploy
+>   6. Modify app to read/query from `date_of_birth_plaintext`, add `date_of_birth` to `User.ignored_columns`
+>   7. Deploy
+>   8. `drop_column :users, :date_of_birth`
+> 2. Create the encrypted column:
+>   1. `create_column :users, :date_of_birth, :enquo_date`
+>   2. Modify the model to write changes to `date_of_birth_plaintext` to `date_of_birth` as well, remove `date_of_birth` from `User.ignored_columns`
+>   3. Deploy
+>   4. Migration to encrypt all `date_of_birth_plaintext` values into `date_of_birth`
+>   5. Deploy
+>   6. Modify app to read/encrypted query from `date_of_birth`, add `date_of_birth_plaintext` to `User.ignored_columns`
+>   7. Deploy
+>   8. `drop_column :users, :date_of_birth_plaintext`

data/e2e_tests/.gitignore

@@ -0,0 +1 @@
+/.pgdbenv

data/e2e_tests/001_direct_migration/exercise_model

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+
+require "active_record"
+require "active_enquo"
+require "pg"
+
+require_relative "../init"
+
+def assert_eq(expected, actual)
+  unless expected == actual
+    $stderr.puts "Expected #{expected.inspect} to equal #{actual.inspect}"
+    exit 1
+  end
+end
+
+def value(f, v)
+  if ENV.key?("USING_ENQUO")
+    People.enquo(f, v)
+  else
+    v
+  end
+end
+
+ActiveRecord::Base.establish_connection(adapter: ENV.fetch("DBTYPE"))
+
+assert_eq(["Meyers"], People.where(first_name: value(:first_name, "Seth")).all.map { |p| p.last_name })
+assert_eq(8, People.where(date_of_birth: value(:date_of_birth, "1980-01-01")..).count)

data/e2e_tests/001_direct_migration/migrations/001_create_people_table.rb

@@ -0,0 +1,11 @@
+class CreatePeopleTable < ActiveRecord::Migration[ENV.fetch("AR_VERSION", "7.0").to_f]
+	def change
+		create_table :people do |t|
+			t.string :first_name
+			t.string :last_name
+			t.date :date_of_birth
+
+			t.timestamps
+		end
+	end
+end

data/e2e_tests/001_direct_migration/migrations/002_encrypt_people_data.rb

@@ -0,0 +1,25 @@
+class EncryptPeopleData < ActiveRecord::Migration[ENV.fetch("AR_VERSION", "7.0").to_f]
+	def up
+		rename_column :people, :first_name, :first_name_plaintext
+		rename_column :people, :last_name, :last_name_plaintext
+		rename_column :people, :date_of_birth, :date_of_birth_plaintext
+
+		add_column :people, :first_name, :enquo_text
+		add_column :people, :last_name, :enquo_text
+		add_column :people, :date_of_birth, :enquo_date
+
+		People.enquo_encrypt_columns(
+			{
+				first_name_plaintext: :first_name,
+				last_name_plaintext: :last_name,
+				date_of_birth_plaintext: :date_of_birth,
+			},
+			# Smol batch size exercises the batching functionality
+			batch_size: 5
+		)
+
+		remove_column :people, :first_name_plaintext
+		remove_column :people, :last_name_plaintext
+		remove_column :people, :date_of_birth_plaintext
+	end
+end

data/e2e_tests/001_direct_migration/run

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+. ../helper.sh
+
+export DBTYPE="postgresql"
+
+check_enquo_pg_db
+clear_pg_db
+
+ar_db_migrate "001"
+load_people
+
+./exercise_model
+
+ar_db_migrate "002"
+USING_ENQUO="y" ./exercise_model

data/e2e_tests/helper.sh

@@ -0,0 +1,34 @@
+if [ -f "../.pgdbenv" ]; then
+	. ../.pgdbenv
+fi
+
+check_enquo_pg_db() {
+	if [ "$(psql -tAc "select count(extname) FROM pg_catalog.pg_extension WHERE extname='pg_enquo'")" != "1" ]; then
+		echo "Specified PostgreSQL database does not have the pg_enquo extension." >&2
+		echo "Check your PG* env vars for correctness, and install the extension if needed." >&2
+		exit 1
+	fi
+}
+
+clear_pg_db() {
+	for tbl in $(psql -tAc "select relname from pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n on n.oid = c.relnamespace where c.relkind='r' and n.nspname = 'public'"); do
+		psql -c "DROP TABLE $tbl" >/dev/null
+	done
+	for seq in $(psql -tAc "select relname from pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n on n.oid = c.relnamespace where c.relkind='S' and n.nspname = 'public'"); do
+		psql -c "DROP SEQUENCE $seq" >/dev/null
+	done
+}
+
+run_ruby() {
+	ruby -r ../init "$@"
+}
+
+ar_db_migrate() {
+	local target_version="$1"
+
+	run_ruby -e "ActiveRecord::MigrationContext.new(['migrations']).up($target_version)" >/dev/null
+}
+
+load_people() {
+	run_ruby -e "People.create!(JSON.parse(File.read('../people.json')))"
+}

data/e2e_tests/init.rb

@@ -0,0 +1,18 @@
+require "active_record"
+require "active_enquo"
+
+ActiveEnquo.root_key = Enquo::RootKey::Static.new("f91c5017a2d946403cc90a688266ff32d186aa2a00efd34dcaa86be802e179d0")
+
+DBTYPE = ENV.fetch("DBTYPE")
+
+case DBTYPE
+when "postgresql"
+	require "pg"
+else
+	raise "Unsupported DBTYPE: #{DBTYPE.inspect}"
+end
+
+class People < ActiveRecord::Base
+end
+
+ActiveRecord::Base.establish_connection(adapter: DBTYPE)

data/e2e_tests/people.json

@@ -0,0 +1 @@
+[{"first_name":"Sergio","last_name":"Leone","date_of_birth":"1929-01-03"},{"first_name":"Bradley","last_name":"Cooper","date_of_birth":"1975-01-05"},{"first_name":"Robert","last_name":"Duvall","date_of_birth":"1931-01-05"},{"first_name":"Kate","last_name":"Moss","date_of_birth":"1974-01-16"},{"first_name":"Laura","last_name":"Schlessinger","date_of_birth":"1947-01-16"},{"first_name":"Jason","last_name":"Segel","date_of_birth":"1980-01-18"},{"first_name":"Pete","last_name":"Buttigieg","date_of_birth":"1982-01-19"},{"first_name":"Mischa","last_name":"Barton","date_of_birth":"1986-01-24"},{"first_name":"Shirley","last_name":"Mason","date_of_birth":"1923-01-25"},{"first_name":"Boris","last_name":"Yeltsin","date_of_birth":"1931-02-01"},{"first_name":"Tom","last_name":"Wilkinson","date_of_birth":"1948-02-05"},{"first_name":"Tony","last_name":"Iommi","date_of_birth":"1948-02-19"},{"first_name":"Sri","last_name":"Srinivasan","date_of_birth":"1967-02-23"},{"first_name":"Tony","last_name":"Randall","date_of_birth":"1920-02-26"},{"first_name":"David","last_name":"Blaine","date_of_birth":"1973-04-04"},{"first_name":"Muddy","last_name":"Waters","date_of_birth":"1915-04-04"},{"first_name":"Danny","last_name":"Almonte","date_of_birth":"1987-04-07"},{"first_name":"Meghann","last_name":"Shaughnessy","date_of_birth":"1979-04-13"},{"first_name":"Maisie","last_name":"Williams","date_of_birth":"1997-04-15"},{"first_name":"Immanuel","last_name":"Kant","date_of_birth":"1724-04-22"},{"first_name":"John","last_name":"Oliver","date_of_birth":"1977-04-23"},{"first_name":"Penelope","last_name":"Cruz","date_of_birth":"1974-04-28"},{"first_name":"Anne","last_name":"Parillaud","date_of_birth":"1961-05-06"},{"first_name":"Cate","last_name":"Blanchett","date_of_birth":"1969-05-14"},{"first_name":"Tucker","last_name":"Carlson","date_of_birth":"1969-05-16"},{"first_name":"Nancy","last_name":"Kwan","date_of_birth":"1939-05-19"},{"first_name":"Melissa","last_name":"Etheridge","date_of_birth":"1961-05-29"},{"first_name":"Marissa","last_name":"Mayer","date_of_birth":"1975-05-30"},{"first_name":"Marvin","last_name":"Hamlisch","date_of_birth":"1944-06-02"},{"first_name":"Gwendolyn","last_name":"Brooks","date_of_birth":"1917-06-07"},{"first_name":"Paul","last_name":"Lynde","date_of_birth":"1927-06-13"},{"first_name":"Paul","last_name":"McCartney","date_of_birth":"1942-06-18"},{"first_name":"Susan","last_name":"Hayward","date_of_birth":"1917-06-30"},{"first_name":"Léa","last_name":"Seydoux","date_of_birth":"1985-07-01"},{"first_name":"Amanda","last_name":"Knox","date_of_birth":"1987-07-09"},{"first_name":"Cindy","last_name":"Sheehan","date_of_birth":"1957-07-10"},{"first_name":"Bill","last_name":"Cosby","date_of_birth":"1937-07-12"},{"first_name":"Raymond","last_name":"Chandler","date_of_birth":"1888-07-23"},{"first_name":"Christopher","last_name":"Nolan","date_of_birth":"1970-07-30"},{"first_name":"Melanie","last_name":"Griffith","date_of_birth":"1957-08-09"},{"first_name":"Freddie","last_name":"Gray","date_of_birth":"1989-08-16"},{"first_name":"Steve","last_name":"Case","date_of_birth":"1958-08-21"},{"first_name":"Cal","last_name":"Ripken","date_of_birth":"1960-08-24"},{"first_name":"Regis","last_name":"Philbin","date_of_birth":"1931-08-25"},{"first_name":"Shania","last_name":"Twain","date_of_birth":"1965-08-28"},{"first_name":"Adam","last_name":"Curry","date_of_birth":"1964-09-03"},{"first_name":"Rachel","last_name":"Hunter","date_of_birth":"1969-09-09"},{"first_name":"Bashar","last_name":"al-Assad","date_of_birth":"1965-09-11"},{"first_name":"Amy","last_name":"Poehler","date_of_birth":"1971-09-16"},{"first_name":"Jim","last_name":"Thompson","date_of_birth":"1906-09-27"},{"first_name":"Matt","last_name":"Damon","date_of_birth":"1970-10-08"},{"first_name":"Leopold","last_name":"Senghor","date_of_birth":"1906-10-09"},{"first_name":"William","last_name":"Penn","date_of_birth":"1644-10-14"},{"first_name":"Rebecca","last_name":"Romijn","date_of_birth":"1971-11-06"},{"first_name":"Auguste","last_name":"Rodin","date_of_birth":"1840-11-12"},{"first_name":"Nate","last_name":"Parker","date_of_birth":"1979-11-18"},{"first_name":"Larry","last_name":"Bird","date_of_birth":"1956-12-07"},{"first_name":"Bobby","last_name":"Flay","date_of_birth":"1964-12-10"},{"first_name":"Chris","last_name":"Evert","date_of_birth":"1954-12-21"},{"first_name":"Frank","last_name":"Zappa","date_of_birth":"1940-12-21"},{"first_name":"Estella","last_name":"Warren","date_of_birth":"1978-12-23"},{"first_name":"Carlos","last_name":"Castaneda","date_of_birth":"1925-12-25"},{"first_name":"Seth","last_name":"Meyers","date_of_birth":"1973-12-28"}]

data/e2e_tests/run

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+for t in [0-9]*; do
+	rv="0"
+	cd "$t"
+
+	./run || rv="$?"
+
+	if [ "$rv" != "0" ]; then
+		echo "Test $i failed with exit code $rv" >&2
+		exit 1
+	fi
+done
+
+echo "All tests passed."

data/lib/active_enquo.rb

@@ -21,7 +21,57 @@ module ActiveEnquo
 	RootKey = Enquo::RootKey
 
 	module ActiveRecord
-		module ModelExtension
+		module QueryFilterMangler
+			class Ciphertext < String
+			end
+			private_constant :Ciphertext
+
+			private
+
+			def mangle_query_filter(a)
+				args = a.first
+				if args.is_a?(Hash)
+					args.keys.each do |attr|
+						next unless enquo_attr?(attr)
+
+						if args[attr].is_a?(Array)
+							args[attr] = args[attr].map { |v| maybe_enquo(attr, v) }
+						elsif args[attr].is_a?(Range)
+							r = args[attr]
+							args[attr] = if r.exclude_end?
+								if r.begin.nil?
+									...maybe_enquo(attr, r.end)
+								elsif r.end.nil?
+									(maybe_enquo(attr.r.begin)...)
+								else
+									maybe_enquo(attr.r.begin)...maybe_enquo(attr, r.end)
+								end
+							else
+								if r.begin.nil?
+									..maybe_enquo(attr, r.end)
+								elsif r.end.nil?
+									maybe_enquo(attr.r.begin)..
+								else
+									maybe_enquo(attr.r.begin)..maybe_enquo(attr, r.end)
+								end
+							end
+						else
+							args[attr] = maybe_enquo(attr, args[attr])
+						end
+					end
+				end
+			end
+
+			def maybe_enquo(attr, v)
+				if v.nil? || v.is_a?(Ciphertext) || v.is_a?(::ActiveRecord::StatementCache::Substitute)
+					v
+				else
+					Ciphertext.new(enquo(attr, v))
+				end
+			end
+		end
+
+		module BaseExtension
 			extend ActiveSupport::Concern
 
 			def _read_attribute(attr_name, &block)
@@ -29,6 +79,7 @@ module ActiveEnquo
 				if t.is_a?(::ActiveEnquo::Type)
 					relation = self.class.arel_table.name
 					value = @attributes.fetch_value(attr_name, &block)
+					return nil if value.nil?
 					field = ::ActiveEnquo.root.field(relation, attr_name)
 					begin
 						t.decrypt(value, @attributes.fetch_value(@primary_key).to_s, field)
@@ -50,10 +101,7 @@ module ActiveEnquo
 					relation = self.class.arel_table.name
 					field = ::ActiveEnquo.root.field(relation, attr_name)
 					attr_opts = self.class.enquo_attribute_options.fetch(attr_name.to_sym, {})
-					safety = if attr_opts[:enable_reduced_security_operations]
-						:unsafe
-					end
-					db_value = t.encrypt(value, @attributes.fetch_value(@primary_key).to_s, field, safety: safety, no_query: attr_opts[:no_query])
+					db_value = t.encrypt(value, @attributes.fetch_value(@primary_key).to_s, field, **attr_opts)
 					@attributes.write_from_user(attr_name, db_value)
 				else
 					super
@@ -61,31 +109,148 @@ module ActiveEnquo
 			end
 
 			module ClassMethods
-				def enquo(attr_name, value)
+				include QueryFilterMangler
+
+				def find_by(*a)
+					mangle_query_filter(a)
+					super
+				end
+
+				def enquo(attr_name, value_or_meta_id, maybe_value = nil)
+					meta_id, value = if value_or_meta_id.is_a?(Symbol)
+						[value_or_meta_id, maybe_value]
+					else
+						[nil, value_or_meta_id]
+					end
+
 					t = self.attribute_types[attr_name.to_s]
 					if t.is_a?(::ActiveEnquo::Type)
 						relation = self.arel_table.name
 						field = ::ActiveEnquo.root.field(relation, attr_name)
-						t.encrypt(value, "", field, safety: :unsafe)
+						if meta_id.nil?
+							t.encrypt(value, "", field, enable_reduced_security_operations: true)
+						else
+							t.encrypt_metadata_value(meta_id, value, field)
+						end
 					else
 						raise ArgumentError, "Cannot produce encrypted value on a non-enquo attribute '#{attr_name}'"
 					end
 				end
 
+				def unenquo(attr_name, value, ctx)
+					t = self.attribute_types[attr_name.to_s]
+					if t.is_a?(::ActiveEnquo::Type)
+						relation = self.arel_table.name
+						field = ::ActiveEnquo.root.field(relation, attr_name)
+						begin
+							t.decrypt(value, ctx, field)
+						rescue Enquo::Error
+							t.decrypt(value, "", field)
+						end
+					else
+						raise ArgumentError, "Cannot decrypt value on a non-enquo attribute '#{attr_name}'"
+					end
+				end
+
+				def enquo_attr?(attr_name)
+					self.attribute_types[attr_name.to_s].is_a?(::ActiveEnquo::Type)
+				end
+
 				def enquo_attr(attr_name, opts)
+					if opts.key?(:default)
+						default_value = opts.delete(:default)
+						after_initialize do
+							next if persisted?
+							next unless self.send(attr_name).nil?
+							self.send(:"#{attr_name}=", default_value.duplicable? ? default_value.dup : default_value)
+						end
+					end
+
 					enquo_attribute_options[attr_name] = @enquo_attribute_options[attr_name].merge(opts)
 				end
 
 				def enquo_attribute_options
 					@enquo_attribute_options ||= Hash.new({})
 				end
+
+				def enquo_encrypt_columns(column_map, batch_size: 10_000)
+					plaintext_columns = column_map.keys
+					relation = self.arel_table.name
+					in_progress = true
+					self.reset_column_information
+
+					while in_progress
+						self.transaction do
+							# The .where("0=1") here is a dummy condition so that the q.or in the .each will work properly
+							q = self.select(self.primary_key).select(plaintext_columns).where("0=1")
+							column_map.each do |pt_col, ct_col|
+								q = q.or(self.where(ct_col => nil).where.not(pt_col => nil))
+							end
+
+							q = q.limit(batch_size).lock
+
+							rows = ::ActiveRecord::Base.connection.exec_query(q.to_sql)
+							if rows.length == 0
+								in_progress = false
+							else
+								rows.each do |row|
+									values = Hash[column_map.map do |pt_col, ct_col|
+										field = ::ActiveEnquo.root.field(relation, ct_col)
+										attr_opts = self.enquo_attribute_options.fetch(ct_col.to_sym, {})
+										t = self.attribute_types[ct_col.to_s]
+										db_value = t.encrypt(row[pt_col.to_s], row[self.primary_key].to_s, field, **attr_opts)
+
+										[ct_col, db_value]
+									end]
+
+									self.where(self.primary_key => row[self.primary_key]).update_all(values)
+								end
+							end
+						end
+					end
+				end
+			end
+		end
+
+		module TableDefinitionExtension
+			def enquo_boolean(name, **options)
+				column(name, :enquo_boolean, **options)
+			end
+
+			def enquo_bigint(name, **options)
+				column(name, :enquo_bigint, **options)
+			end
+
+			def enquo_date(name, **options)
+				column(name, :enquo_date, **options)
+			end
+
+			def enquo_text(name, **options)
+				column(name, :enquo_text, **options)
 			end
 		end
+
+		module RelationExtension
+			include QueryFilterMangler
+			extend ActiveSupport::Concern
+
+			def where(*a)
+				mangle_query_filter(a)
+				super
+			end
+
+			def exists?(*a)
+				mangle_query_filter(a)
+				super
+			end
+
+		end
 	end
 
 	module Postgres
 		module ConnectionAdapter
 			def initialize_type_map(m = type_map)
+				m.register_type "enquo_boolean", ActiveEnquo::Type::Boolean.new
 				m.register_type "enquo_bigint", ActiveEnquo::Type::Bigint.new
 				m.register_type "enquo_date", ActiveEnquo::Type::Date.new
 				m.register_type "enquo_text", ActiveEnquo::Type::Text.new
@@ -96,13 +261,35 @@ module ActiveEnquo
 	end
 
 	class Type < ::ActiveRecord::Type::Value
+		class Boolean < Type
+			def type
+				:enquo_boolean
+			end
+
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_boolean(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
+			end
+
+			def decrypt(value, context, field)
+				field.decrypt_boolean(value, context)
+			end
+		end
+
 		class Bigint < Type
 			def type
 				:enquo_bigint
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
-				field.encrypt_i64(value, context, safety: safety, no_query: no_query)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_i64(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
 			end
 
 			def decrypt(value, context, field)
@@ -115,9 +302,13 @@ module ActiveEnquo
 				:enquo_date
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false)
 				value = cast_to_date(value)
-				field.encrypt_date(value, context, safety: safety, no_query: no_query)
+				if value.nil?
+					value
+				else
+					field.encrypt_date(value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query)
+				end
 			end
 
 			def decrypt(value, context, field)
@@ -131,6 +322,8 @@ module ActiveEnquo
 					value
 				elsif value.respond_to?(:to_date)
 					value.to_date
+				elsif value.nil? || (value.respond_to?(:empty?) && value.empty?) || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					nil
 				else
 					Time.parse(value.to_s).to_date
 				end
@@ -142,26 +335,73 @@ module ActiveEnquo
 				:enquo_text
 			end
 
-			def encrypt(value, context, field, safety: true, no_query: false)
-				field.encrypt_text(value, context, safety: safety, no_query: no_query)
+			def encrypt(value, context, field, enable_reduced_security_operations: false, no_query: false, enable_ordering: false)
+				if enable_ordering && !enable_reduced_security_operations
+					raise ArgumentError, "Cannot enable ordering on an Enquo attribute unless Reduced Security Operations are enabled"
+				end
+
+				if value.nil? || value.is_a?(::ActiveRecord::StatementCache::Substitute)
+					value
+				else
+					field.encrypt_text(value.respond_to?(:encode) ? value.encode("UTF-8") : value, context, safety: enable_reduced_security_operations ? :unsafe : true, no_query: no_query, order_prefix_length: enable_ordering ? 8 : nil)
+				end
+			end
+
+			def encrypt_metadata_value(name, value, field)
+				case name
+				when :length
+					field.encrypt_text_length_query(value)
+				else
+					raise ArgumentError, "Unknown metadata name for Text field: #{name.inspect}"
+				end
 			end
 
 			def decrypt(value, context, field)
-				field.decrypt_text(value, context)
+				if value.nil?
+					nil
+				else
+					field.decrypt_text(value, context)
+				end
+			end
+		end
+	end
+
+	if defined?(Rails::Railtie)
+		class Initializer < Rails::Railtie
+			initializer "active_enquo.root_key" do |app|
+				if app
+					if app.credentials
+						if app.credentials.active_enquo
+							if root_key = app.credentials.active_enquo.root_key
+								ActiveEnquo.root_key = Enquo::RootKey::Static.new(root_key)
+							else
+								Rails.logger.warn "Could not initialize ActiveEnquo, as no active_enquo.root_key credential was found for this environment"
+							end
+						else
+							Rails.logger.warn "Could not initialize ActiveEnquo, as no active_enquo credentials were found for this environment"
+						end
+					else
+						Rails.logger.warn "Could not initialize ActiveEnquo, as no credentials were found for this environment"
+					end
+				else
+					Rails.logger.warn "Could not initialize ActiveEnquo, as no app was found for this environment"
+				end
 			end
 		end
 	end
 end
 
 ActiveSupport.on_load(:active_record) do
-	::ActiveRecord::Base.send :include, ActiveEnquo::ActiveRecord::ModelExtension
+	::ActiveRecord::Relation.prepend ActiveEnquo::ActiveRecord::RelationExtension
+	::ActiveRecord::Base.include ActiveEnquo::ActiveRecord::BaseExtension
+
+	::ActiveRecord::ConnectionAdapters::Table.include ActiveEnquo::ActiveRecord::TableDefinitionExtension
+	::ActiveRecord::ConnectionAdapters::TableDefinition.include ActiveEnquo::ActiveRecord::TableDefinitionExtension
 
 	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend ActiveEnquo::Postgres::ConnectionAdapter
-#	::ActiveRecord::Type.register(:enquo_bigint, ActiveEnquo::Type::Bigint, adapter: :postgresql)
 
-	unless ActiveRecord::VERSION::MAJOR == 7
-		::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_bigint] = {}
-		::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_date] = {}
-		::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_text] = {}
-	end
+	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_boolean] = { name: "enquo_boolean" }
+	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_bigint] = { name: "enquo_bigint" }
+	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_date]   = { name: "enquo_date" }
+	::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:enquo_text]   = { name: "enquo_text" }
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: active_enquo
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.5.0
 platform: ruby
 authors:
 - Matt Palmer
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-10-07 00:00:00.000000000 Z
+date: 2023-04-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: enquo-core
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0.7'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0.7'
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -195,6 +195,16 @@ files:
 - README.md
 - active_enquo.gemspec
 - docs/DEVELOPMENT.md
+- docs/MIGRATION.md
+- e2e_tests/.gitignore
+- e2e_tests/001_direct_migration/exercise_model
+- e2e_tests/001_direct_migration/migrations/001_create_people_table.rb
+- e2e_tests/001_direct_migration/migrations/002_encrypt_people_data.rb
+- e2e_tests/001_direct_migration/run
+- e2e_tests/helper.sh
+- e2e_tests/init.rb
+- e2e_tests/people.json
+- e2e_tests/run
 - lib/active_enquo.rb
 homepage: https://enquo.org
 licenses: []