witsec 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 4ac7324ae9480a4bbabaca4d462bb4205a2fa470b1f872c7226f3a809c2c588f
+  data.tar.gz: 1d2fa60ac0531ad1f010f74658f7a6aeadddd0f343fc4712f9cc2f0caeb097d2
+SHA512:
+  metadata.gz: d580f3f361aaac94331a2935c7ed79da3e82106925f1c0012bc6b6e5319de1e77e1a28e295a8c47eb0ab42c8cec56a11f69b1363034397633fbdf37964750a56
+  data.tar.gz: c46f2392c59e03d4b81ab353c4e83b1c928eaf7974b21e3e6e55c70707fc33b3b9a5937d2990fd8bb47d2a69d3a3b5b481c777b2e5454472d8efe52de58f4e9b

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright Nicolai Bach Woller
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,87 @@
+# Witsec
+When developing Rails applications you end up with a large difference between the size of the database used in development and the database in production. This makes it hard to gauge, how the performance is impacted as the amount of data grows.
+
+You can try to generate a large set of data, but there is not guarantee, that the data you generate will produce the same performance issues as the real data would. Another approach is to download a database dump, but then you have real production data lying on your machine - including any sensitive data like SSNs or addresses.
+
+This gem tries to avoid this by copying all data to a new database, anonymizing it all in the process. This new database can now be dumped and used in development.
+
+## Installation
+Add this line to your application's Gemfile:
+
+```ruby
+gem "witsec"
+# if you want a simple way to generate fake data
+gem "faker"
+```
+
+And then execute:
+```bash
+$ bundle
+```
+Then create a new file at `config/witsec/schema.rb`. This path will become configurable in a later version.
+Finally configure your database to create a database to store the anonymized data in:
+```yaml
+production:
+  primary:
+    <<: *default
+    host: your_host_url
+    database: you_app_name
+  anonymized:
+    <<: *default
+    host: some_url_that_might_or_might_not_be_the_same_as_your_host_url
+    database: anonymized # This must be called anonymized for now. It will become configurable in a later version. 
+    migrations_paths: db/migrate
+```
+
+## Usage
+Witsec uses a schema file to determine what to anonymize and how to do it.
+
+```ruby
+# config/witsec/schema.rb
+Witsec::Schema.define(2025_01_15_142512) do
+  anonymize_table "addresses" do |t|
+    t.column "street", using: -> { Faker::Address.street_address }
+    t.column "zip_code", using: -> { Faker::Address.zip_code }
+    t.column "city", using: "New York"
+  end
+
+  include_table "animals"
+
+  exclude_table "government_secrets"
+end
+```
+`Witsec::Schema.define` requires an integer param. This should match the latest timestamp in your app's `db/schema.rb` and is used to ensure, that you have considered any changes introduced in database migrations. A warning is shown if a mismatch is detected, when you run the `bin/rails witsec:schema:verify` task. An error will be raised in a later version, when attempting to anonymize a database with a mismatch in versions.
+
+There are three ways to anonymize a table:
+
+#### `anonymize_table`
+Takes the name of a table to be anonymized and a block, determining how each column should be masked. In the example above, [Faker](https://github.com/faker-ruby/faker) is used to provide a random address, but you can put whatever you want in the lambda or even provide a static value as is done on the city column.
+
+Any column not mentioned in the block, will **not** be anonymized.
+
+#### `include_table` 
+Takes the name of a table to be copied in its entirety without any masking at all. Use this for tables without **any** sensitive data.
+
+#### `exclude_table`
+Takes the name of a table to be excluded. No data will be copied. If any other tables reference anything in an excluded table, you are probably going to have a bad time.
+
+### Rake tasks
+Witsec comes with some tasks for anonymizing the database and verifying that the schema is up to date.
+
+#### `witsec:anonymize`
+Anonymizes the app's primary database using the configuration defined in your schema.
+
+#### `witsec:scheme:verify_tables`
+Checks that the tables in your database are all mentioned in your Witsec schema. Useful as a step in your CI.
+
+#### `witsec:scheme:verify_version`
+Checks that your Witsec::Schema version matches the version of your latest run migration. Useful as a step in your CI.
+
+#### `witsec:scheme:verify`
+Runs all other verifications
+
+## Contributing
+Bug reports and pull requests are welcome on GitHub at https://github.com/traels-it/witsec.
+
+## License
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,3 @@
+require "bundler/setup"
+
+require "bundler/gem_tasks"

data/lib/tasks/witsec_tasks.rake

@@ -0,0 +1,71 @@
+namespace :witsec do
+  desc "Anonymizes the app's primary database"
+  task anonymize: :environment do
+    Witsec::Anonymizer.new.anonymize
+  end
+
+  namespace :schema do
+    desc "Verify that the Witsec schema defines an anonymization for each table"
+    task :verify_tables, [:raise_error] => :environment do |_t, args|
+      args.with_defaults(raise_error: false)
+
+      anonymizer = Witsec::Anonymizer.new
+
+      if anonymizer.schema.table_names != ActiveRecord::Base.connection.tables.sort
+        missing_tables = ActiveRecord::Base.connection.tables - anonymizer.schema.table_names
+        extra_tables = anonymizer.schema.table_names - ActiveRecord::Base.connection.tables
+
+        messages = ["Witsec schema contains errors"]
+
+        messages << "Missing tables: #{missing_tables.join(", ")}" if missing_tables.any?
+        messages << "Extra tables: #{extra_tables.join(", ")}" if extra_tables.any?
+
+        return unless messages.any?
+
+        if args[:raise_error]
+          raise Witsec::TableMismatchError, messages.join("\n")
+        else
+          abort messages.join("\n")
+        end
+      end
+    end
+
+    desc "Verify the Witsec schema version matches the version of the latest run migration"
+    task :verify_version, [:raise_error] => :environment do |_t, args|
+      args.with_defaults(raise_error: false)
+
+      app_schema_version = ActiveRecord::Base.connection.schema_version
+      witsec_schema_version = Witsec::Anonymizer.new.schema.version
+      message = "Witsec schema version (#{witsec_schema_version}) does not match app's schema version (#{app_schema_version})"
+
+      if app_schema_version != witsec_schema_version
+        if args[:raise_error]
+          raise Witsec::VersionError, message
+        else
+          abort message
+        end
+      end
+    end
+
+    desc "Runs all Witsec verifications"
+    task verify: :environment do
+      error_messages = []
+      begin
+        Rake::Task["witsec:schema:verify_tables"].invoke(true)
+      rescue Witsec::TableMismatchError => error
+        error_messages << error.message
+      end
+      begin
+        Rake::Task["witsec:schema:verify_version"].invoke(true)
+      rescue Witsec::VersionError => error
+        error_messages << error.message
+      end
+
+      if error_messages.any?
+        abort error_messages.join("\n")
+      else
+        puts "Witsec schema is all good 🎉"
+      end
+    end
+  end
+end

data/lib/witsec/alias.rb

@@ -0,0 +1,29 @@
+module Witsec
+  class Alias
+    def initialize(table_name, columns:, schema:)
+      @table_name = table_name
+      @columns = columns
+      @schema = schema
+    end
+
+    def anonymize(rows)
+      rows.map do |row|
+        table = schema.anonymized_tables.find { _1.name == table_name }
+
+        columns.each_with_index.map do |column, index|
+          anonymized_column, mask = table.columns.find { |name, _mask| name == column }
+
+          if anonymized_column.present?
+            mask.respond_to?(:call) ? mask.call : mask
+          else
+            row[index]
+          end
+        end
+      end
+    end
+
+    private
+
+    attr_reader :table_name, :columns, :schema
+  end
+end

data/lib/witsec/anonymizer.rb

@@ -0,0 +1,93 @@
+module Witsec
+  class Anonymizer
+    BATCH_SIZE = 1000
+
+    def initialize
+      @schema = instance_eval(File.read("config/witsec/schema.rb"))
+    end
+
+    attr_reader :schema
+
+    # TODO: Make silence configurable
+    def anonymize
+      time = Benchmark.measure do
+        ActiveRecord::Base.logger.silence do
+          clear_output_database
+
+          ActiveRecord::Base.connection.tables.each do |table_name|
+            if schema.anonymizes?(table_name)
+              # A performance improvement could probably be found here, if we just passed along included tables (as in tables, where no rows are anonymized) without querying etc.
+
+              input_connection = input_connection_pool.lease_connection
+              record_rows = input_connection.execute("SELECT * FROM #{table_name}").to_a
+              columns = record_rows&.first&.keys
+              rows = record_rows.map(&:values)
+              puts "Anonymizing #{table_name} (#{rows.size} rows)"
+              input_connection_pool.release_connection
+
+              anonymized_rows = Witsec::Alias.new(table_name, columns:, schema:).anonymize(rows)
+              output_connection = output_connection_pool.lease_connection
+              # If referential integrity is not disabled, you have to create all rows in the correct order
+              output_connection.disable_referential_integrity do
+                # Use insert for performance
+                row_batches = anonymized_rows.in_groups_of(BATCH_SIZE, false)
+                total = 0
+                row_batches.each_with_index do |batch, index|
+                  print "Anonymizing up to row #{total + batch.size} of #{rows.size}\r"
+                  total += batch.size
+                  values = batch.map do |row|
+                    "(#{row.map { |value| ActiveRecord::Base.connection.quote(value) }.join(", ")})"
+                  end.join(", ")
+
+                  output_connection.execute(
+                    "INSERT INTO #{table_name} (#{columns.join(", ")}) VALUES #{values}"
+                  )
+                end
+              end
+
+              output_connection_pool.release_connection
+            else
+              puts "Skipping #{table_name}"
+              next
+            end
+          end
+        end
+      end
+      puts "Anonymized all in #{time.real} seconds"
+    end
+
+    def clear_output_database
+      puts "Clearing output database"
+
+      ActiveRecord::Base.logger.silence do
+        connection = output_connection_pool.lease_connection
+
+        connection.disable_referential_integrity do
+          connection.tables.each do |table_name|
+            connection.execute("TRUNCATE TABLE #{table_name} CASCADE")
+          end
+        end
+
+        output_connection_pool.release_connection
+      end
+    end
+
+    private
+
+    def input_database_configuration
+      Rails.configuration.database_configuration[Rails.env]["primary"]
+    end
+
+    def input_connection_pool
+      ActiveRecord::Base.establish_connection(input_database_configuration)
+    end
+
+    def output_database_configuration
+      Rails.configuration.database_configuration[Rails.env]["anonymized"]
+    end
+
+    def output_connection_pool
+      ActiveRecord::Base.establish_connection(output_database_configuration)
+    end
+  end
+end

data/lib/witsec/railtie.rb

@@ -0,0 +1,9 @@
+module Witsec
+  class Railtie < ::Rails::Railtie
+    rake_tasks do
+      Dir.glob(File.join(File.dirname(__FILE__), "../tasks/**/*.rake")).each do |rake_file|
+        load rake_file
+      end
+    end
+  end
+end

data/lib/witsec/schema.rb

@@ -0,0 +1,73 @@
+module Witsec
+  class Schema
+    class << self
+      def define(version, &block)
+        new(version).define(&block)
+      end
+    end
+
+    def initialize(version)
+      @version = version
+      @anonymized_tables = []
+      @excluded_tables = []
+    end
+
+    attr_reader :version, :anonymized_tables, :excluded_tables
+
+    def define(&block)
+      instance_eval(&block)
+
+      self
+    end
+
+    def exclude_table(name)
+      excluded_tables << name
+    end
+
+    def include_table(name)
+      anonymized_tables << Table.new(name)
+    end
+
+    def anonymize_table(name, &block)
+      anonymized_tables << Table.new(name).define do
+        instance_eval(&block)
+      end
+    end
+
+    def anonymizes?(table_name)
+      anonymized_table_names.include?(table_name)
+    end
+
+    def table_names
+      (anonymized_table_names + excluded_tables).sort
+    end
+
+    private
+
+    def anonymized_table_names
+      anonymized_tables.map(&:name)
+    end
+  end
+
+  class Table
+    def initialize(name)
+      @name = name
+      @columns = []
+    end
+
+    attr_reader :name, :columns
+
+    def define(&block)
+      instance_eval(&block)
+
+      self
+    end
+
+    def column(column_name, using: nil)
+      columns << [column_name, using]
+    end
+  end
+
+  TableMismatchError = Class.new(StandardError)
+  VersionError = Class.new(StandardError)
+end

data/lib/witsec/version.rb

@@ -0,0 +1,3 @@
+module Witsec
+  VERSION = "0.1.0"
+end

data/lib/witsec.rb

@@ -0,0 +1,9 @@
+require "witsec/version"
+require "witsec/railtie"
+require "witsec/alias"
+require "witsec/anonymizer"
+require "witsec/schema"
+
+module Witsec
+  # Your code goes here...
+end

metadata

@@ -0,0 +1,108 @@
+--- !ruby/object:Gem::Specification
+name: witsec
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Nicolai Bach Woller
+bindir: bin
+cert_chain: []
+date: 2025-02-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '8.0'
+- !ruby/object:Gem::Dependency
+  name: minitest-spec-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: faker
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: standard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.44'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.44'
+email:
+- woller@traels.it
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- Rakefile
+- lib/tasks/witsec_tasks.rake
+- lib/witsec.rb
+- lib/witsec/alias.rb
+- lib/witsec/anonymizer.rb
+- lib/witsec/railtie.rb
+- lib/witsec/schema.rb
+- lib/witsec/version.rb
+homepage: https://github.com/traels-it/witsec
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/traels-it/witsec
+  source_code_uri: https://github.com/traels-it/witsec
+  changelog_uri: https://github.com/traels-it/witsec/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.2
+specification_version: 4
+summary: Anonymize your database for dumping
+test_files: []