polo 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3cc4c8cc8d6e2ecfb65d39c7ec4c49192c643388
-  data.tar.gz: efdec09f03b61b966650144674799234e2c10369
+  metadata.gz: 241a2fd317d2b766497c31f824c4a9c76309df49
+  data.tar.gz: cdfa23f93211a11f43414aba198b9edcf9e6203a
 SHA512:
-  metadata.gz: 10a1d5a28d71cfda25e83d108231dd2daa068075bdff5f356c37270039c090ddb3652246c5223a4bd3e1a3d30d2ce71a6e47961e7d73eecce4892ecf6fc55e5b
-  data.tar.gz: af6ebc5a5f4f08cf7106ad53a24244e5b8cc1b0dab4c7cd988d88134e8357a888bee464627b589095e0d08f0b218f091e59af02cd7ef592a1c7f43196fee6a4a
+  metadata.gz: 63c36ebd40006d1f542474e86f4f91d343b1b99ab6b59054597f749274b3bdc919ea9ea2a5db8bfac2eafe46123030b55084ce2039767907b68555867acc3585
+  data.tar.gz: 4765c393aa6e1b19b784bcbc8d14dc22e3fcab1f8a7a048bb1dfdf01e3f269290a51957f271aead96e1fbedf24de87aab70949dd14018a010aa451932a97e4d6

data/CHANGELOG.md

@@ -0,0 +1,35 @@
+## 0.2.0
+
+### Breaking Changes
+
+- None
+
+### Added
+
+- [#8](https://github.com/IFTTT/polo/pull/8) Global settings
+- [#17](https://github.com/IFTTT/polo/pull/17) Using random generator instead of character shuffle for data obfuscation
+- [#18](https://github.com/IFTTT/polo/pull/18) Add a CHANGELOG
+- [#20](https://github.com/IFTTT/polo/pull/20) Postgres Support
+
+### Fixed
+
+- Typo fixes on the README: [#9](https://github.com/IFTTT/polo/pull/9), [#10](https://github.com/IFTTT/polo/pull/10)
+- [#11]() Some ActiveRecord classes do not use id as the primary key
+- [#25](https://github.com/IFTTT/polo/pull/25) Fix Custom Strategy
+
+## 0.1.0
+
+### Breaking Changes
+
+- None
+
+### Added
+
+- [#2](https://github.com/IFTTT/polo/pull/2) Add :ignore and :override options to deal with data collision
+- [#3](https://github.com/IFTTT/polo/pull/3) Add option to obfuscate fields
+- [#4](https://github.com/IFTTT/polo/pull/4) Add intro to Update / Ignore section
+- [#6](https://github.com/IFTTT/polo/pull/6) Set up Appraisal to run specs across Rails 3.2 through 4.2
+
+### Fixed
+
+- [#7](https://github.com/IFTTT/polo/pull/7) Fix casting of values

data/README.md

@@ -1,4 +1,5 @@
 [![Open Source at IFTTT](http://ifttt.github.io/images/open-source-ifttt.svg)](http://ifttt.github.io)
+[![Build Status](https://travis-ci.org/IFTTT/polo.svg?branch=master)](https://travis-ci.org/IFTTT/polo)
 
 ![Polo](https://raw.githubusercontent.com/IFTTT/polo/images/images/polo.png "Polo")
 
@@ -9,6 +10,9 @@ Polo takes an `ActiveRecord::Base` seed object and traverses every whitelisted `
 
 You can then save those SQL `INSERTS` to .sql file and import the data to your favorite environment.
 
+# Motivation
+Read our [blog post](https://medium.com/engineering-at-ifttt/happier-rails-development-with-polo-9df6819136d3#.f8ll3azeq) or check out this [presentation](https://speakerdeck.com/nettofarah/polo-working-with-real-world-data-in-development).
+
 ## Usage
 Given the following data model:
 ```ruby
@@ -74,7 +78,8 @@ INSERT INTO `ingredients` (`id`, `name`, `quantity`) VALUES (4, 'Cheese', '2 sli
 
 Occasionally, you might have a dataset that you want to refresh. A production database that has data that might be useful on your local copy of the database. Polo doesn't have an opinion about your data; if you try to import data with a key that's already in your local database, Polo doesn't necessarily know how you want to handle that conflict.
 
-Advanced users will find the `on_duplicate` option to be helpful in this context. It gives Polo instructions on how to handle collisions. *Note: This feature is currently only supported for MySQL databases. (PRs for other databases are welcome!)*
+Advanced users will find the `on_duplicate` option to be helpful in this context. It gives Polo instructions on how to handle collisions.
+*Note: This feature is currently only supported for MySQL databases. (PRs for other databases are welcome!)*
 
 There are two possible values for the `on_duplicate` key: `:ignore` and `:override`. Ignore keeps the old data. Override keeps the new data. If there's a collision and the on_duplicate param is not set, Polo will simpy stop importing the data.
 
@@ -128,9 +133,26 @@ Polo::Traveler.explore(AR::Chef, 1)
 INSERT INTO `chefs` (`id`, `name`, `email`) VALUES (1, 'Netto', 'eahorctmaagfo.nitm@l')
 ```
 
-Warning: This is not a security feature. Fields can still easily be rearranged back to their original format. Polo will simply scramble the order of strings so you don't accidentaly end up causing side effects when using production data in development.
+Warning: This is not a security feature. Fields can still easily be rearranged back to their original format. Polo will simply scramble the order of strings so you don't accidentally end up causing side effects when using production data in development. It is not a good practice to use highly sensitive data in development.
+
+#### Advanced Obfuscation
+
+For more advanced obfuscation, you can pass in a custom obfuscation strategy.
+Polo will take in a lambda that can be used to transform sensitive data.
+
+````ruby
+Polo.configure do
+  email_strategy = lambda {|e| "#{e.split("@")[0]}_test@example.com" }
+  credit_card_strategy = lambda {|_| "4111111111111111"}
+  obfuscate({email: email_strategy, credit_card: credit_card_strategy})
+end
+
+Polo::Traveler.explore(AR::Chef, 1)
+````
 
-It is not a good practice to use highly sensitive data in development.
+```sql
+INSERT INTO `chefs` (`id`, `name`, `email`) VALUES (1, 'Netto', 'netto_test.@example.com')
+```
 
 ## Installation
 
@@ -152,7 +174,7 @@ $ gem install polo
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/polo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/IFTTT/polo. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.
 
 To run the specs across all supported version of Rails, check out the repo and
 follow these steps:

data/lib/polo/collector.rb

@@ -14,10 +14,12 @@ module Polo
     # ActiveSupport::Notifications block and collecting every generate SQL query.
     #
     def collect
-      ActiveSupport::Notifications.subscribed(collector, 'sql.active_record') do
-        base_finder = @base_class.includes(@dependency_tree).where(id: @id)
-        collect_sql(@base_class, base_finder.to_sql)
-        base_finder.to_a
+      unprepared_statement do
+        ActiveSupport::Notifications.subscribed(collector, 'sql.active_record') do
+          base_finder = @base_class.includes(@dependency_tree).where(@base_class.primary_key => @id)
+          collect_sql(@base_class, base_finder.to_sql)
+          base_finder.to_a
+        end
       end
 
       @selects.compact.uniq
@@ -33,8 +35,9 @@ module Polo
     #
     def collector
       lambda do |name, start, finish, id, payload|
+        return unless payload[:name] =~ /^(.*) Load$/
         begin
-          class_name = payload[:name].gsub(' Load', '').constantize
+          class_name = $1.constantize
           sql = payload[:sql]
           collect_sql(class_name, sql)
         rescue ActiveRecord::StatementInvalid, NameError
@@ -49,5 +52,15 @@ module Polo
         sql: sql
       }
     end
+
+    def unprepared_statement
+      if ActiveRecord::Base.connection.respond_to?(:unprepared_statement)
+        ActiveRecord::Base.connection.unprepared_statement do
+          yield
+        end
+      else
+        yield
+      end
+    end
   end
 end

data/lib/polo/configuration.rb

@@ -4,13 +4,30 @@ module Polo
     attr_reader :on_duplicate_strategy, :blacklist
 
     def initialize(options={})
-      options = { on_duplicate: nil, obfuscate: [] }.merge(options)
+      options = { on_duplicate: nil, obfuscate: {} }.merge(options)
       @on_duplicate_strategy = options[:on_duplicate]
-      @blacklist = options[:obfuscate]
+      obfuscate(options[:obfuscate])
     end
 
+    # TODO: document this
+    # This normalizes an array or hash of fields to a hash of
+    # { field_name => strategy }
     def obfuscate(*fields)
-      @blacklist = fields
+      if fields.is_a?(Array)
+        fields = fields.flatten
+      end
+
+      fields_and_strategies = {}
+
+      fields.each do |field|
+        if field.is_a?(Symbol) || field.is_a?(String)
+          fields_and_strategies[field] = nil
+        elsif field.is_a?(Hash)
+          fields_and_strategies = fields_and_strategies.merge(field)
+        end
+      end
+
+      @blacklist = fields_and_strategies
     end
 
     def on_duplicate(strategy)

data/lib/polo/translator.rb

@@ -25,7 +25,7 @@ module Polo
       end
 
       if fields = @configuration.blacklist
-        obfuscate!(active_record_instances, fields.map(&:to_s))
+        obfuscate!(active_record_instances, fields)
       end
 
       active_record_instances
@@ -35,12 +35,25 @@ module Polo
 
     def obfuscate!(instances, fields)
       instances.each do |instance|
-        next if (instance.attributes.keys & fields).empty?
-        fields.each do |field|
-          value = instance.attributes[field] || ''
-          instance.send("#{field}=", value.split('').shuffle.join)
+        next if intersection(instance.attributes.keys, fields).empty?
+
+        fields.each do |field, strategy|
+          value = instance.attributes[field.to_s] || ''
+          instance.send("#{field}=", new_field_value(field, strategy, value))
         end
       end
     end
+
+    def intersection(attrs, fields)
+      attrs & fields.keys.map(&:to_s)
+    end
+
+    def new_field_value(field, strategy, value)
+      if strategy.nil?
+        value.split("").shuffle.join
+      else
+        strategy.call(value)
+      end
+    end
   end
 end

data/lib/polo/version.rb

@@ -1,3 +1,3 @@
 module Polo
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: polo
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Netto Farah
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2015-09-14 00:00:00.000000000 Z
+date: 2015-12-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -107,6 +107,7 @@ files:
 - ".ruby-version"
 - ".travis.yml"
 - Appraisals
+- CHANGELOG.md
 - CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE.txt