polo 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 241a2fd317d2b766497c31f824c4a9c76309df49
-  data.tar.gz: cdfa23f93211a11f43414aba198b9edcf9e6203a
+  metadata.gz: d3f754e19121b0fb417f692c5ee333daeb8de049
+  data.tar.gz: 326da5077810478da02f66de52e749cac05866cc
 SHA512:
-  metadata.gz: 63c36ebd40006d1f542474e86f4f91d343b1b99ab6b59054597f749274b3bdc919ea9ea2a5db8bfac2eafe46123030b55084ce2039767907b68555867acc3585
-  data.tar.gz: 4765c393aa6e1b19b784bcbc8d14dc22e3fcab1f8a7a048bb1dfdf01e3f269290a51957f271aead96e1fbedf24de87aab70949dd14018a010aa451932a97e4d6
+  metadata.gz: 453c79f229642062534266178aabc162bbfdf06231c2b0237c828ef9277cc96c9987a9471096664a06c16b313f058c95f68e194bdd824a79ec01f232ad3a55f1
+  data.tar.gz: bfa5e063eb14430036621742fb3bc72e5159cf307a3efbd2e09894fb93c95c5face35766e8e78a7496ae9d2c7f05349796c021cd5452319271af0a688ea2a5a5

data/.gitignore

@@ -7,3 +7,4 @@
 /pkg/
 /spec/reports/
 /tmp/
+/gemfiles/*.gemfile.lock

data/.ruby-version

@@ -1 +1 @@
-2.1.5
+2.2.1

data/.travis.yml

@@ -6,4 +6,4 @@ gemfile:
   - gemfiles/rails40.gemfile
   - gemfiles/rails41.gemfile
   - gemfiles/rails42.gemfile
-before_install: gem install bundler -v 1.10.6
+before_install: gem install bundler -v 1.11.2

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 0.3.0
+
+### Breaking Changes
+
+- None
+
+### Added
+
+- [#30](https://github.com/IFTTT/polo/pull/30) Advanced obfuscation
+- [#37](https://github.com/IFTTT/polo/pull/37) Custom adapters for Postgres and MySQL
+
+### Fixed
+
+- [#26](https://github.com/IFTTT/polo/pull/26) Postgres - Use ActiveRecord methods to generate INSERT SQLs
+- [#25](https://github.com/IFTTT/polo/pull/25) Fix custom strategies bug
+- [#28](https://github.com/IFTTT/polo/pull/28) Only obfuscate fields when they are present
+- [#35](https://github.com/IFTTT/polo/pull/35) Better support for Rails 4.0
+- [#31](https://github.com/IFTTT/polo/pull/31) Fix link to Code of Conduct
+
 ## 0.2.0
 
 ### Breaking Changes

data/README.md

@@ -10,7 +10,7 @@ Polo takes an `ActiveRecord::Base` seed object and traverses every whitelisted `
 
 You can then save those SQL `INSERTS` to .sql file and import the data to your favorite environment.
 
-# Motivation
+## Motivation
 Read our [blog post](https://medium.com/engineering-at-ifttt/happier-rails-development-with-polo-9df6819136d3#.f8ll3azeq) or check out this [presentation](https://speakerdeck.com/nettofarah/polo-working-with-real-world-data-in-development).
 
 ## Usage
@@ -137,14 +137,26 @@ Warning: This is not a security feature. Fields can still easily be rearranged b
 
 #### Advanced Obfuscation
 
-For more advanced obfuscation, you can pass in a custom obfuscation strategy.
-Polo will take in a lambda that can be used to transform sensitive data.
+For more advanced obfuscation, you can pass in a custom obfuscation strategy. Polo will take in a lambda that can be used to transform sensitive data. 
+
+Using a `:symbol` as an obfuscate key targets all columns of that name. Passing an SQL selector as a `String` will target columns within the specified table.
 
 ````ruby
 Polo.configure do
-  email_strategy = lambda {|e| "#{e.split("@")[0]}_test@example.com" }
-  credit_card_strategy = lambda {|_| "4111111111111111"}
-  obfuscate({email: email_strategy, credit_card: credit_card_strategy})
+
+  email_strategy = lambda do |email|
+    first_part = email.split("@")[0]
+    "#{first_part}@test.com"
+  end
+  
+  credit_card_strategy = lambda do |credit_card|
+    "4123 4567 8910 1112"
+  end
+  
+  obfuscate({
+    'chefs.email' => email_strategy, # This only applies to the "email" column in the "chefs" table
+    :credit_card  => credit_card_strategy # This applies to any column named "credit_card" across every table
+  })
 end
 
 Polo::Traveler.explore(AR::Chef, 1)
@@ -174,7 +186,7 @@ $ gem install polo
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/IFTTT/polo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/IFTTT/polo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Code of Conduct](https://github.com/IFTTT/polo/blob/master/CODE_OF_CONDUCT.md).
 
 To run the specs across all supported version of Rails, check out the repo and
 follow these steps:

data/lib/polo/adapters/mysql.rb

@@ -0,0 +1,24 @@
+module Polo
+  module Adapters
+    class MySQL
+      def on_duplicate_key_update(inserts, records)
+        insert_and_record = inserts.zip(records)
+        insert_and_record.map do |insert, record|
+          values_syntax = record.attributes.keys.map do |key|
+            "#{key} = VALUES(#{key})"
+          end
+
+          on_dup_syntax = "ON DUPLICATE KEY UPDATE #{values_syntax.join(', ')}"
+
+          "#{insert} #{on_dup_syntax}"
+        end
+      end
+
+      def ignore_transform(inserts, records)
+        inserts.map do |insert|
+          insert.gsub("INSERT", "INSERT IGNORE")
+        end
+      end
+    end
+  end
+end

data/lib/polo/adapters/postgres.rb

@@ -0,0 +1,32 @@
+module Polo
+  module Adapters
+    class Postgres
+      # TODO: Implement UPSERT. This command became available in 9.1.
+      #
+      # See: http://www.the-art-of-web.com/sql/upsert/
+      def on_duplicate_key_update(inserts, records)
+        raise 'on_duplicate: :override is not currently supported in the PostgreSQL adapter'
+      end
+
+      # Internal: Transforms an INSERT with PostgreSQL-specific syntax. Ignores
+      #           records that alread exist in the table. To do this, it uses
+      #           a heuristic, i.e. checks if there is a record with the same id
+      #           in the table.
+      #           See: http://stackoverflow.com/a/6527838/32816
+      #
+      # inserts - The Array of INSERT statements.
+      # records - The Array of Arel objects.
+      #
+      # Returns the Array of transformed INSERT statements.
+      def ignore_transform(inserts, records)
+        insert_and_record = inserts.zip(records)
+        insert_and_record.map do |insert, record|
+          table_name = record.class.arel_table.name
+          id = record[:id]
+          insert = insert.gsub(/VALUES \((.+)\)$/m, 'SELECT \\1')
+          insert << " WHERE NOT EXISTS (SELECT 1 FROM #{table_name} WHERE id=#{id});"
+        end
+      end
+    end
+  end
+end

data/lib/polo/configuration.rb

@@ -1,10 +1,11 @@
 module Polo
 
   class Configuration
-    attr_reader :on_duplicate_strategy, :blacklist
+    attr_reader :on_duplicate_strategy, :blacklist, :adapter
 
     def initialize(options={})
-      options = { on_duplicate: nil, obfuscate: {} }.merge(options)
+      options = { on_duplicate: nil, obfuscate: {}, adapter: :mysql }.merge(options)
+      @adapter = options[:adapter]
       @on_duplicate_strategy = options[:on_duplicate]
       obfuscate(options[:obfuscate])
     end
@@ -33,5 +34,9 @@ module Polo
     def on_duplicate(strategy)
       @on_duplicate_strategy = strategy
     end
+
+    def set_adapter(db)
+      @adapter = db
+    end
   end
 end

data/lib/polo/sql_translator.rb

@@ -1,93 +1,89 @@
 require 'active_record'
 require 'polo/configuration'
+require 'polo/adapters/mysql'
+require 'polo/adapters/postgres'
 
 module Polo
   class SqlTranslator
 
-    def initialize(object, configuration=Configuration.new)
+    def initialize(object, configuration = Configuration.new)
       @record = object
       @configuration = configuration
-    end
-
-    def to_sql
-      records = Array.wrap(@record)
 
-      sqls = records.map do |record|
-        raw_sql(record)
+      case @configuration.adapter
+      when :mysql
+        @adapter = Polo::Adapters::MySQL.new
+      when :postgres
+        @adapter = Polo::Adapters::Postgres.new
+      else
+        raise "Unknown SQL adapter: #{@configuration.adapter}"
       end
-
-      if @configuration.on_duplicate_strategy == :ignore
-        sqls = ignore_transform(sqls)
-      end
-
-      if @configuration.on_duplicate_strategy == :override
-        sqls = on_duplicate_key_update(sqls, records)
-      end
-
-      sqls
     end
 
-    private
-
-    def on_duplicate_key_update(sqls, records)
-      insert_and_record = sqls.zip(records)
-      insert_and_record.map do |insert, record|
-        values_syntax = record.attributes.keys.map do |key|
-          "#{key} = VALUES(#{key})"
-        end
-
-        on_dup_syntax = "ON DUPLICATE KEY UPDATE #{values_syntax.join(', ')}"
-
-        "#{insert} #{on_dup_syntax}"
+    def to_sql
+      case @configuration.on_duplicate_strategy
+      when :ignore
+        @adapter.ignore_transform(inserts, records)
+      when :override
+        @adapter.on_duplicate_key_update(inserts, records)
+      else inserts
       end
     end
 
-    def ignore_transform(inserts)
-      inserts.map do |insert|
-        insert.gsub("INSERT", "INSERT IGNORE")
-      end
+    def records
+      Array.wrap(@record)
     end
 
-    def raw_sql(record)
-      connection = ActiveRecord::Base.connection
-      attributes = record.attributes
-
-      keys = attributes.keys.map do |key|
-        "`#{key}`"
+    def inserts
+      records.map do |record|
+        raw_sql(record)
       end
+    end
 
-      values = attributes.map do |key, value|
-        column = record.column_for_attribute(key)
-        connection.quote(cast_attribute(record, column, value))
-      end
+    private
 
-      "INSERT INTO `#{record.class.table_name}` (#{keys.join(', ')}) VALUES (#{values.join(', ')})"
+    # Internal: Generates an insert SQL statement for a given record
+    #
+    # It will make use of the InsertManager class from the Arel gem to generate
+    # insert statements
+    #
+    def raw_sql(record)
+      record.class.arel_table.create_insert.tap do |insert_manager|
+        insert_manager.insert(insert_values(record))
+      end.to_sql
     end
 
-    module ActiveRecordLessThanFourPointTwo
-      def cast_attribute(record, column, value)
-        attribute = record.send(:type_cast_attribute_for_write, column, value)
-
-        if record.class.serialized_attributes.include?(column.name)
-          attribute.serialize
-        else
-          attribute
-        end
+    # Internal: Returns an object's attribute definitions along with
+    # their set values (for Rails 3.x).
+    #
+    module ActiveRecordLessThanFour
+      def insert_values(record)
+        record.send(:arel_attributes_values)
       end
     end
 
-    module ActiveRecordFourPointTwoOrGreater
-      def cast_attribute(record, column, value)
-        column.type_cast_for_database(value)
+    # Internal: Returns an object's attribute definitions along with
+    # their set values (for Rails >= 4.x).
+    #
+    # From Rails 4.2 onwards, for some reason attributes with custom serializers
+    # wouldn't be properly serialized automatically. That's why explict
+    # 'type_cast' call are necessary.
+    #
+    module ActiveRecordFourOrGreater
+      def insert_values(record)
+        connection = ActiveRecord::Base.connection
+        values = record.send(:arel_attributes_with_values_for_create, record.attribute_names)
+        values.each do |attribute, value|
+          column = record.send(:column_for_attribute, attribute.name)
+          values[attribute] = connection.type_cast(value, column)
+        end
       end
     end
 
-    if ActiveRecord::VERSION::STRING.start_with?('3.2') ||
-        ActiveRecord::VERSION::STRING.start_with?('4.0') ||
-        ActiveRecord::VERSION::STRING.start_with?('4.1')
-      include ActiveRecordLessThanFourPointTwo
+    if ActiveRecord::VERSION::MAJOR < 4
+      include ActiveRecordLessThanFour
     else
-      include ActiveRecordFourPointTwoOrGreater
+      include ActiveRecordFourOrGreater
     end
   end
 end

data/lib/polo/translator.rb

@@ -38,14 +38,31 @@ module Polo
         next if intersection(instance.attributes.keys, fields).empty?
 
         fields.each do |field, strategy|
-          value = instance.attributes[field.to_s] || ''
-          instance.send("#{field}=", new_field_value(field, strategy, value))
+          field = field.to_s
+
+          if table = table_name(field)
+            field = field_name(field)
+          end
+
+          correct_table = table.nil? || instance.class.table_name == table
+
+          if correct_table && value = instance.attributes[field]
+            instance.send("#{field}=", new_field_value(field, strategy, value))
+          end
         end
       end
     end
 
+    def field_name(field)
+      field.to_s.include?('.') ? field.split('.').last : field.to_s
+    end
+
+    def table_name(field)
+      field.to_s.include?('.') ? field.split('.').first : nil
+    end
+
     def intersection(attrs, fields)
-      attrs & fields.keys.map(&:to_s)
+      attrs & fields.map { |pair| field_name(pair.first) }
     end
 
     def new_field_value(field, strategy, value)

data/lib/polo/version.rb

@@ -1,3 +1,3 @@
 module Polo
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: polo
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Netto Farah
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-12-04 00:00:00.000000000 Z
+date: 2016-04-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -114,14 +114,12 @@ files:
 - README.md
 - Rakefile
 - gemfiles/rails32.gemfile
-- gemfiles/rails32.gemfile.lock
 - gemfiles/rails40.gemfile
-- gemfiles/rails40.gemfile.lock
 - gemfiles/rails41.gemfile
-- gemfiles/rails41.gemfile.lock
 - gemfiles/rails42.gemfile
-- gemfiles/rails42.gemfile.lock
 - lib/polo.rb
+- lib/polo/adapters/mysql.rb
+- lib/polo/adapters/postgres.rb
 - lib/polo/collector.rb
 - lib/polo/configuration.rb
 - lib/polo/sql_translator.rb

data/gemfiles/rails32.gemfile.lock

@@ -1,61 +0,0 @@
-PATH
-  remote: ../
-  specs:
-    polo (0.1.0)
-      activerecord (>= 3.2)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activemodel (3.2.22)
-      activesupport (= 3.2.22)
-      builder (~> 3.0.0)
-    activerecord (3.2.22)
-      activemodel (= 3.2.22)
-      activesupport (= 3.2.22)
-      arel (~> 3.0.2)
-      tzinfo (~> 0.3.29)
-    activesupport (3.2.22)
-      i18n (~> 0.6, >= 0.6.4)
-      multi_json (~> 1.0)
-    appraisal (2.1.0)
-      bundler
-      rake
-      thor (>= 0.14.0)
-    arel (3.0.3)
-    builder (3.0.4)
-    diff-lcs (1.2.5)
-    i18n (0.7.0)
-    multi_json (1.11.2)
-    rake (10.4.2)
-    rspec (3.3.0)
-      rspec-core (~> 3.3.0)
-      rspec-expectations (~> 3.3.0)
-      rspec-mocks (~> 3.3.0)
-    rspec-core (3.3.2)
-      rspec-support (~> 3.3.0)
-    rspec-expectations (3.3.1)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-mocks (3.3.2)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-support (3.3.0)
-    sqlite3 (1.3.10)
-    thor (0.19.1)
-    tzinfo (0.3.44)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activerecord (= 3.2.22)
-  appraisal
-  bundler (~> 1.10)
-  polo!
-  rake (~> 10.0)
-  rspec
-  sqlite3 (= 1.3.10)
-
-BUNDLED WITH
-   1.10.6

data/gemfiles/rails40.gemfile.lock

@@ -1,67 +0,0 @@
-PATH
-  remote: ../
-  specs:
-    polo (0.1.0)
-      activerecord (>= 3.2)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activemodel (4.0.13)
-      activesupport (= 4.0.13)
-      builder (~> 3.1.0)
-    activerecord (4.0.13)
-      activemodel (= 4.0.13)
-      activerecord-deprecated_finders (~> 1.0.2)
-      activesupport (= 4.0.13)
-      arel (~> 4.0.0)
-    activerecord-deprecated_finders (1.0.4)
-    activesupport (4.0.13)
-      i18n (~> 0.6, >= 0.6.9)
-      minitest (~> 4.2)
-      multi_json (~> 1.3)
-      thread_safe (~> 0.1)
-      tzinfo (~> 0.3.37)
-    appraisal (2.1.0)
-      bundler
-      rake
-      thor (>= 0.14.0)
-    arel (4.0.2)
-    builder (3.1.4)
-    diff-lcs (1.2.5)
-    i18n (0.7.0)
-    minitest (4.7.5)
-    multi_json (1.11.2)
-    rake (10.4.2)
-    rspec (3.3.0)
-      rspec-core (~> 3.3.0)
-      rspec-expectations (~> 3.3.0)
-      rspec-mocks (~> 3.3.0)
-    rspec-core (3.3.2)
-      rspec-support (~> 3.3.0)
-    rspec-expectations (3.3.1)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-mocks (3.3.2)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-support (3.3.0)
-    sqlite3 (1.3.10)
-    thor (0.19.1)
-    thread_safe (0.3.5)
-    tzinfo (0.3.44)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activerecord (= 4.0.13)
-  appraisal
-  bundler (~> 1.10)
-  polo!
-  rake (~> 10.0)
-  rspec
-  sqlite3 (= 1.3.10)
-
-BUNDLED WITH
-   1.10.6

data/gemfiles/rails41.gemfile.lock

@@ -1,66 +0,0 @@
-PATH
-  remote: ../
-  specs:
-    polo (0.1.0)
-      activerecord (>= 3.2)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activemodel (4.1.12)
-      activesupport (= 4.1.12)
-      builder (~> 3.1)
-    activerecord (4.1.12)
-      activemodel (= 4.1.12)
-      activesupport (= 4.1.12)
-      arel (~> 5.0.0)
-    activesupport (4.1.12)
-      i18n (~> 0.6, >= 0.6.9)
-      json (~> 1.7, >= 1.7.7)
-      minitest (~> 5.1)
-      thread_safe (~> 0.1)
-      tzinfo (~> 1.1)
-    appraisal (2.1.0)
-      bundler
-      rake
-      thor (>= 0.14.0)
-    arel (5.0.1.20140414130214)
-    builder (3.2.2)
-    diff-lcs (1.2.5)
-    i18n (0.7.0)
-    json (1.8.3)
-    minitest (5.8.0)
-    rake (10.4.2)
-    rspec (3.3.0)
-      rspec-core (~> 3.3.0)
-      rspec-expectations (~> 3.3.0)
-      rspec-mocks (~> 3.3.0)
-    rspec-core (3.3.2)
-      rspec-support (~> 3.3.0)
-    rspec-expectations (3.3.1)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-mocks (3.3.2)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-support (3.3.0)
-    sqlite3 (1.3.10)
-    thor (0.19.1)
-    thread_safe (0.3.5)
-    tzinfo (1.2.2)
-      thread_safe (~> 0.1)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activerecord (= 4.1.12)
-  appraisal
-  bundler (~> 1.10)
-  polo!
-  rake (~> 10.0)
-  rspec
-  sqlite3 (= 1.3.10)
-
-BUNDLED WITH
-   1.10.6

data/gemfiles/rails42.gemfile.lock

@@ -1,66 +0,0 @@
-PATH
-  remote: ../
-  specs:
-    polo (0.1.0)
-      activerecord (>= 3.2)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activemodel (4.2.3)
-      activesupport (= 4.2.3)
-      builder (~> 3.1)
-    activerecord (4.2.3)
-      activemodel (= 4.2.3)
-      activesupport (= 4.2.3)
-      arel (~> 6.0)
-    activesupport (4.2.3)
-      i18n (~> 0.7)
-      json (~> 1.7, >= 1.7.7)
-      minitest (~> 5.1)
-      thread_safe (~> 0.3, >= 0.3.4)
-      tzinfo (~> 1.1)
-    appraisal (2.1.0)
-      bundler
-      rake
-      thor (>= 0.14.0)
-    arel (6.0.3)
-    builder (3.2.2)
-    diff-lcs (1.2.5)
-    i18n (0.7.0)
-    json (1.8.3)
-    minitest (5.8.0)
-    rake (10.4.2)
-    rspec (3.3.0)
-      rspec-core (~> 3.3.0)
-      rspec-expectations (~> 3.3.0)
-      rspec-mocks (~> 3.3.0)
-    rspec-core (3.3.2)
-      rspec-support (~> 3.3.0)
-    rspec-expectations (3.3.1)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-mocks (3.3.2)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-support (3.3.0)
-    sqlite3 (1.3.10)
-    thor (0.19.1)
-    thread_safe (0.3.5)
-    tzinfo (1.2.2)
-      thread_safe (~> 0.1)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activerecord (= 4.2.3)
-  appraisal
-  bundler (~> 1.10)
-  polo!
-  rake (~> 10.0)
-  rspec
-  sqlite3 (= 1.3.10)
-
-BUNDLED WITH
-   1.10.6