ferry 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 135d110be1a1feb5802cb1035af0fab9bf98bea1
+  data.tar.gz: b53e81709fc2a3bbef1314732f8feae82ffe2d41
+SHA512:
+  metadata.gz: 8274f3a35429634000afd5e6cd106ee24e92a6db2ff7a4e40db9ebd80768ca9f743e1c3534757b23cdd8a161ad1177a7564f118961a6eec083831e61b03629f3
+  data.tar.gz: d9f0d4414f852a925b20e0ee9697a5bec2b203b8974ac6018e440f1f195b28014d474e51fa6c2e11d449c40ff06622f5a9eb0b1d534595f5c74a8004b7c68d25

data/.gitignore

@@ -0,0 +1,22 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in ferry.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Anthony Corletti
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,155 @@
+# Ferry
+
+Ferry is a data migration and data manipulation tool that seeks to quickly and easily reduce overhead when dealing with big data problems.
+
+## TO-DO
+
+- [ ] Refactoring before public release
+  - [x] Define action-items for refactor
+  - [x] Provide working example(s) of using ferry
+  - [ ] Public release fine-tuning
+- [ ] Tests
+  - [ ] Testing input for migrate method (max_workers, batch_size)
+  - [ ] Testing that there is an ActiveRecord::Relation object being passed to find_in_batches
+  - [ ] Migration Scenarios - dummy class migration
+- [ ] Refactor logging logic into Logger class
+  - [x] Initial revision
+  - [ ] Review
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'ferry'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install ferry
+
+## Usage
+
+Usage pending. See examples / submit PR's for your ideas.
+
+## Example(s)
+
+###### 29 July 2014
+Version 0.0.1 is functional with the rake task defined here :: https://github.com/customink/design_content_migration/blob/master/lib/tasks/ferry_example.rake#L10
+
+Please manually install ferry from your locally cloned repo ...
+```
+git clone git@github.com:customink/ferry.git
+cd ferry
+gem build ferry.gemspec
+gem install ferry
+```
+add it to your app's Gemfile
+```
+gem 'ferry'
+```
+and then
+```
+bundle install
+```
+as it has not been pushed to rubygems.com yet.
+
+Tests - Coming soon to an editor near me!
+
+###### 28 July 2014
+Ferry should not support Oracle.
+
+###### 25 July 2014
+After a few more reviews with @metaskills, @gilr00y, @jdlehman, and @danielwheeler1987, Ferry will extend ActiveRecord with a "migrate" (more legit name search still in naming progress) method. From there we are going to pass the same relation to find in batches to a worker which will plow through the batch passed to it via a yield call from the task.
+
+Tests will include; validate the data passed into the worker (log) and testing that there is an ActiveRecord::Relation being passed to find_in_batches.
+
+###### 23 July 2014
+After a few chats with @gilr00y and @jdlehman Ferry may extend ActiveRecord with a "migrate" method we could call on an ActiveRecord object. From there that object would call an Engine instance with appropriate fields to kickoff the actual data migration.
+
+There is some logic duplication and layer duplication between the Engine class and the "migrate" method that extends ActiveRecord. Still working out how to concisely write logic that handles the management of forking connection and engine init calls.
+
+```
+require "ferry/version"
+require 'models/engine'
+require 'models/logger'
+
+module Ferry
+  class ActiveRecord
+    def self.migrate(&block)
+      yield
+    end
+  end
+end
+```
+
+This implementation should be able to run something like this ...
+
+```
+engine = Engine.new(
+  Design.where("savedate > ?", 6.months.ago.strftime("%d.%m.%Y %H").to_datetime).id,
+  Design.where("savedate > ?", 3.months.ago.strftime("%d.%m.%Y %H").to_datetime).id,
+  100_000,
+  1_000,
+  "log/ferry"
+)
+
+Design.where("savedate > ?", 130.hours.ago.strftime("%d.%m.%Y %H").to_datetime).migrate(
+  engine.run do | start_id, end_id, chunk_size, batch_size, log |
+    worker.run do | start_id, chunk_size, batch_size, log |
+      worker_end_id = start_id + chunk_size - 1
+      Design.where("id >= ? && id <= ?", start_id, worker_end_id).find_in_batches(batch_size: batch_size) do |batch|
+        # move and manipulate data as you please
+      end
+      start_id += batch_size
+    end
+  end
+)
+
+```
+
+###### 22 July 2014
+After installing ferry to your local machine or bundling from your gemfile - in your migration task make sure to define your chunker as such ...
+
+```
+require 'ferry'
+
+namespace :example do
+  task "my_migration_task" do
+
+    ferry = Engine.new(
+      :max_workers  => number_of_workers ex:8,
+      :start_id     => where_are_we_starting ex:2910, Model.first.id,
+      :end_id       => where_are_we_ending ex:8190, Model.last.id,
+      :chunk_size   => size_of_chunks_that_workers_will_process ex:42,
+      :working_dir  => ex:"path/to/working_dir"
+    )
+
+    ferry.run do |start_id, chunk_size, log|
+      begin
+        work = Model.select(":id").where("? <= id and id < ?", start_id, start_id + chunk_size)
+        rows_to_process = rel.count
+        log.puts("rows_to_process: #{rows_to_process}")
+        work.find_in_batches(:batch_size => 1_000) do
+          # doing things and logging stuff as you please ...
+        end
+      rescue Exception => e
+        log.puts "Broken on id #{id}"
+        raise e
+      end
+    end
+
+  end
+end
+```
+
+
+## Contributing
+
+1. Fork it ( https://github.com/[my-github-username]/ferry/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+Bundler::GemHelper.install_tasks

data/ferry.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'ferry/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "ferry"
+  spec.version       = Ferry::VERSION
+  spec.authors       = ["Anthony Corletti", "Logan Watanabe", "Larry Heimann"]
+  spec.email         = ["anthcor@gmail.com", "loganwatanabe@gmail.com", "profh@cmu.edu"]
+  spec.summary       = "Ferry is a data migration and data manipulation tool"
+  spec.description   = "Ferry is a data migration and data manipulation tool that seeks to simplify the increasingly prevalent big data problems that tech companies face"
+  spec.homepage      = "https://github.com/cmu-is-projects/"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "activerecord"
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "progressbar"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "minitest"
+end

data/lib/ferry.rb

@@ -0,0 +1,7 @@
+require "ferry/version"
+require "ferry/engine"
+require "ferry/logger"
+
+module Ferry
+  # Your code goes here...
+end

data/lib/ferry/engine.rb

@@ -0,0 +1,16 @@
+class Engine
+  def initialize(options={})
+  end
+
+  def run(options, &block)
+    log = options[:log]
+    collection = options[:batch]
+    log.write "collection length: #{collection.length}"
+    begin
+      instance_exec(collection, &block)
+    rescue Exception => e
+      log.write "Error: #{e}"
+    end
+    log.write "worker finished"
+  end
+end

data/lib/ferry/logger.rb

@@ -0,0 +1,12 @@
+class Logger
+  def initialize(options={})
+    @homedir = options[:homedir] ||= "log"
+    FileUtils.mkdir @homedir unless Dir[@homedir].present?
+    FileUtils.touch "#{@homedir}/ferry.log"
+  end
+
+  def write(msg)
+    log = File.open("#{@homedir}/ferry.log", 'w')
+    log.puts msg
+  end
+end

data/lib/ferry/version.rb

@@ -0,0 +1,3 @@
+module Ferry
+  VERSION = "0.0.1"
+end

metadata

@@ -0,0 +1,129 @@
+--- !ruby/object:Gem::Specification
+name: ferry
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Anthony Corletti
+- Logan Watanabe
+- Larry Heimann
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-08-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: progressbar
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Ferry is a data migration and data manipulation tool that seeks to simplify
+  the increasingly prevalent big data problems that tech companies face
+email:
+- anthcor@gmail.com
+- loganwatanabe@gmail.com
+- profh@cmu.edu
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- ferry.gemspec
+- lib/ferry.rb
+- lib/ferry/engine.rb
+- lib/ferry/logger.rb
+- lib/ferry/version.rb
+homepage: https://github.com/cmu-is-projects/
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Ferry is a data migration and data manipulation tool
+test_files: []