mobilize-base 1.351 → 1.361

data/README.md

@@ -1,675 +1,4 @@
 Mobilize
 ========
 
-Mobilize is an end-to-end data transfer workflow manager with:
-* a Google Spreadsheets UI through [google-drive-ruby][google_drive_ruby];
-* a queue manager through [Resque][resque];
-* a persistent caching / database layer through [Mongoid][mongoid];
-* gems for data transfers to/from Hive, mySQL, and HTTP endpoints
-  (coming soon).
-
-Mobilize-Base includes all the core scheduling and processing
-functionality, allowing you to:
-* put workers on the Mobilize Resque queue.
-* create [Users](#section_Start_Users_User) and their associated Google Spreadsheet [Runners](#section_Start_Users_Runner);
-* poll for [Jobs](#section_Job) on Runners (currently gsheet to gsheet only) and add them to Resque;
-* monitor the status of Jobs on a rolling log.
-
-Table Of Contents
------------------
-* [Overview](#section_Overview)
-* [Install](#section_Install)
-  * [Redis](#section_Install_Redis)
-  * [MongoDB](#section_Install_MongoDB)
-  * [Mobilize-Base](#section_Install_Mobilize-Base)
-  * [Default Folders and Files](#section_Install_Folders_and_Files)
-* [Configure](#section_Configure)
-  * [Google Drive](#section_Configure_Google_Drive)
-  * [Google Sheets](#section_Configure_Google_Sheets)
-  * [Jobtracker](#section_Configure_Jobtracker)
-  * [Resque](#section_Configure_Resque)
-  * [Resque-Web](#section_Configure_Resque-Web)
-  * [Gridfs](#section_Configure_Gridfs)
-  * [Mongoid](#section_Configure_Mongoid)
-* [Start](#section_Start)
-  * [Start Resque-Web](#section_Start_Start_Resque-Web)
-  * [Set Environment](#section_Start_Set_Environment)
-  * [Create User](#section_Start_Create_User)
-  * [Start Workers](#section_Start_Start_Workers)
-  * [View Logs](#section_Start_View_Logs)
-  * [Start Jobtracker](#section_Start_Start_Jobtracker)
-  * [Create Job](#section_Start_Create_Job)
-  * [Run Test](#section_Start_Run_Test)
-  * [Add Gbooks and Gsheets](#section_Start_Add_Gbooks_And_Gsheets)
-* [Meta](#section_Meta)
-* [Author](#section_Author)
-* [Special Thanks](#section_Special_Thanks)
-
-
-<a name='section_Overview'></a>
-Overview
------------
-
-* Mobilize is a script deployment and data visualization framework with
-a Google Spreadsheets UI.
-* Mobilize uses Resque for parallelization and queueuing, MongoDB for caching,
-and Google Drive for hosting, user input and display.
-* The [mobilize-ssh][mobilize-ssh] gem allows you to run scripts and
-copy files between different machines, and have output directed to a
-spreadsheet for viewing and processing.
-* The platform is easily extensible: add your own rake tasks and
-handlers by following a few simple conventions, and you can have your own
-Mobilize gem up and running in no time.
-
-<a name='section_Install'></a>
-Install
-------------
-
-Mobilize requires Ruby 1.9.3, and has been tested on OSX and Ubuntu.
-
-[RVM][rvm] is great for managing your rubies. 
-
-<a name='section_Install_Redis'></a>
-### Redis
-
-Redis is a pre-requisite for running Resque. 
-
-Please refer to the [Resque Redis Section][redis] for complete
-instructions.
-
-<a name='section_Install_MongoDB'></a>
-### MongoDB
-
-MongoDB is used to persist caches between reads and writes, keep track
-of Users and Jobs, and store Datasets that map to endpoints.
-
-Please refer to the [MongoDB Quickstart Page][mongodb_quickstart] to get started.
-
-The settings for database and port are set in config/mongoid.yml
-and are best left as default. Please refer to [Configure
-Mongoid](#section_Configure_Mongoid) for details.
-
-<a name='section_Install_Mobilize-Base'></a>
-### Mobilize-Base
-
-Mobilize-Base contains all of the gems it needs to run. 
-
-add this to your Gemfile:
-
-``` ruby
-gem "mobilize-base"
-```
-
-or do
-
-  $ gem install mobilize-base
-
-for a ruby-wide install.
-
-<a name='section_Install_Folders_and_Files'></a>
-### Folders and Files
-
-Mobilize requires a config folder and a log folder. 
-
-If you're on Rails, it will use the built-in config and log folders. 
-
-Otherwise, it will use log and config folders in the project folder (the
-same one that contains your Rakefile)
-
-### Rakefile
-
-Inside the Rakefile in your project's root folder, make sure you have:
-
-``` ruby
-require 'mobilize-base/tasks'
-```
-
-This defines rake tasks essential to run the environment.
-
-### Config and Log Folders
-
-run 
-
-  $ rake mobilize_base:setup
-
-Mobilize will create config/mobilize/ and log/ folders at the project root
-level. (same as the Rakefile). 
-
-(You can override these by passing
-MOBILIZE_CONFIG_DIR and/or MOBILIZE_LOG_DIR arguments to the command.
-All directories must end with a '/'.)
-
-The script will also create samples for all required config files, which are detailed below.
-
-Resque will create a mobilize-resque-`<environment>`.log in the log folder,
-and loop over 10 files, 10MB each.
-
-<a name='section_Configure'></a>
-Configure
-------------
-
-All Mobilize configurations live in files in `config/mobilize/*.yml` by
-default. Samples can
-be found below or on github in the [lib/samples][git_samples] folder.
-
-<a name='section_Configure_Google_Drive'></a>
-### Configure Google Drive
-
-gdrive.yml needs:
-* a domain, which can be gmail.com but may be different depending on
-your organization. All gdrive accounts should have
-the same domain, and all Users should have emails in this domain.
-* an owner name and password. You can set up separate owners
-  for different environments as in the below file, which will keep your
-mission critical workers from getting rate-limit errors.
-* one admin_group_name, which the owner and all admins should be added to -- this
-group will need read permissions to read from and edit permissions to write
-to files.
-* one or more admins with email attributes -- these will be for people
-  who should be given write permissions to all Mobilize books in the
-environment for maintenance purposes.
-* one worker_group_name, which the owner and all workers should be added to -- this
-group will need read permissions to read from and edit permissions to write
-to files.
-* one or more workers with name and pw attributes -- they will be used
-  to queue up google reads and writes. This can be the same as the owner
-account for testing purposes or low-volume environments. 
-
-__Mobilize only allows one Resque
-worker at a time to use a Google drive worker account for
-reading/writing, which is called a gdrive_slot.__
-
-Sample gdrive.yml:
-
-``` yml
----
-development:
-  domain: host.com
-  owner:
-    name: owner_development
-    pw: google_drive_password
-  admin_group_name: admins_development
-  admins:
-    - name: admin
-  worker_group_name: workers_development
-  workers:
-    - name: worker_development001 
-      pw: worker001_google_drive_password
-    - name: worker_development002, 
-      pw: worker002_google_drive_password
-test:
-  domain: host.com
-  owner:
-    name: owner_test
-    pw: google_drive_password
-  admin_group_name: admins_test
-  admins:
-    - name: admin
-  worker_group_name: workers_test
-  workers:
-    - name: worker_test001 
-      pw: worker001_google_drive_password
-    - name: worker_test002 
-      pw: worker002_google_drive_password
-production:
-  domain: host.com
-  owner:
-    name: owner_production
-    pw: google_drive_password
-  admin_group_name: admins_production
-  admins:
-    - name: admin
-  worker_group_name: workers_production
-  workers:
-    - name: worker_production001 
-      pw: worker001_google_drive_password
-    - name: worker_production002
-      pw: worker002_google_drive_password
-```
-
-<a name='section_Configure_Google_Sheets'></a>
-### Configure Google Sheets
-
-gsheet.yml needs:
-* max_cells, which is the number of cells a sheet is allowed to have
-  written to it at one time. Default is 50k cells, which is about how
-much you can write before things start breaking.
-* Because Google Docs ties date formatting to the Locale for the
-  spreadsheet, there are 2 date format parameters:
-  * read_date_format, which is the format that should be read FROM google
-    sheets for date columns.
-  * sheet_date_format, which is the format that the google sheet is in.
-  * A date column is defined as one where the column header = "date" or "Date", or ends with "_date" or "Date".
-  * The defaults are set to US locale for sheet_date_format, because in 'Murica (US) we
-    use %m/%d/%Y for some reason, and to %Y-%m-%d format for
-    reading, which is more standard and sorts well as a string. If your
-    locale is NOT 'Murica you will want to change these.
-
-Sample gsheet.yml
-
-``` yml
----
-development:
-  max_cells: 400000
-  read_date_format: "%Y-%m-%d"
-  sheet_date_format: "%m/%d/%Y"
-test:
-  max_cells: 400000
-  read_date_format: "%Y-%m-%d"
-  sheet_date_format: "%m/%d/%Y"
-staging:
-  max_cells: 400000
-  read_date_format: "%Y-%m-%d"
-  sheet_date_format: "%m/%d/%Y"
-production:
-  max_cells: 400000
-  read_date_format: "%Y-%m-%d"
-  sheet_date_format: "%m/%d/%Y"
-```
-
-<a name='section_Configure_Jobtracker'></a>
-### Configure Jobtracker
-
-The Jobtracker sits on your Resque and does 2 things:
-* check for Users that are due for polling;
-* send out notifications when:
-  * there are failed jobs on Resque;
-  * there are jobs on Resque that have run beyond the max run time.
-
-Emails are sent using ActionMailer, through the owner Google Drive
-account.
-  * errors are sent to the owner of the job/stage as well as the admin group.
-  * errors not specific to a job/stage are sent to the admin group only, as
-    given in gdrive.yml
-
-To this end, it needs these parameters, for which there is a sample
-below and in the [lib/samples][git_samples] folder:
-
-``` yml
----
-development:
-  cycle_freq: 10 #time between Jobtracker sweeps
-  notification_freq: 3600 #1 hour between failure/timeout notifications
-  runner_read_freq: 300 #5 min between runner reads
-  max_run_time: 14400 # if a job runs for 4h+, notification will be sent
-  extensions: [] #additional Mobilize modules to load workers with
-test:
-  cycle_freq: 10 #time between Jobtracker sweeps
-  notification_freq: 3600 #1 hour between failure/timeout notifications
-  runner_read_freq: 300 #5 min between runner reads
-  max_run_time: 14400 # if a job runs for 4h+, notification will be sent
-  extensions: [] #additional Mobilize modules to load workers with
-production:
-  cycle_freq: 10 #time between Jobtracker sweeps
-  notification_freq: 3600 #1 hour between failure/timeout notifications
-  runner_read_freq: 300 #5 min between runner reads
-  max_run_time: 14400 # if a job runs for 4h+, notification will be sent
-  extensions: [] #additional Mobilize modules to load workers with
-```
-
-<a name='section_Configure_Resque'></a>
-### Configure Resque
-
-Resque keeps track of Jobs, Workers and logging.
-
-It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
-
-* queue_name - the name of the Resque queue where you would like the Jobtracker and Resque Workers to
-  run. Default is mobilize.
-* max_workers - the total number of simultaneous workers you would like
-  on your queue. Default is 4 for development and test, 36 in
-production, but feel free to adjust depending on your hardware.
-* redis_port - you should probably leave this alone, it specifies the
-  default port for dev and prod and a separate one for testing.
-* web_port - this specifies the port under which resque-web operates
-
-``` yml
----
-development:
-  queue_name: mobilize
-  max_workers: 4
-  redis_port: 6379
-  web_port: 8282
-test:
-  queue_name: mobilize
-  max_workers: 4
-  redis_port: 9736
-  web_port: 8282
-production:
-  queue_name: mobilize
-  max_workers: 36
-  redis_port: 6379
-  web_port: 8282
-```
-
-<a name='section_Configure_Resque-Web'></a>
-### Configure Resque-Web
-
-Please change your default username and password in the resque_web.rb
-file in your config folder, reproduced below:
-
-``` ruby
-#comment out the below if you want no authentication on your web portal (not recommended)
-Resque::Server.use(Rack::Auth::Basic) do |user, password|
-  [user, password] == ['admin', 'changeyourpassword']
-end
-```
-
-This file is passed as a config file argument to
-mobilize_base:resque_web task, as detailed in [Start Resque-Web](#section_Start_Start_Resque-Web).
-
-<a name='section_Configure_Gridfs'></a>
-### Configure Gridfs
-
-Mobilize stores cached data in MongoDB Gridfs. 
-It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
-
-* max_compressed_write_size - the amount of compressed data Gridfs will
-allow. If you try to write more than this, an exception will be thrown.
-
-``` yml
----
-development:
-  max_compressed_write_size: 1000000000 #~1GB
-test:
-  max_compressed_write_size: 1000000000 #~1GB
-production:
-  max_compressed_write_size: 1000000000 #~1GB
-```
-
-<a name='section_Configure_Mongoid'></a>
-### Configure Mongoid
-
-Mongoid is the abstraction layer on top of MongoDB so we can interact
-with it in an ActiveRecord-like fashion. 
-
-It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
-
-You shouldn't need to change anything in this file.
-
-``` yml
----
-development:
-  sessions:
-    default:
-      database: mobilize-development
-      persist_in_safe_mode: true
-      hosts:
-        - 127.0.0.1:27017
-test:
-  sessions:
-    default:
-      database: mobilize-test
-      persist_in_safe_mode: true
-      hosts:
-        - 127.0.0.1:27017
-production:
-  sessions:
-    default:
-      database: mobilize-production
-      persist_in_safe_mode: true
-      hosts:
-        - 127.0.0.1:27017
-```
-
-<a name='section_Start'></a>
-Start
------
-
-A Mobilize instance can be considered "started" or "running" when you have:
-
-1. Resque workers running on the Mobilize queue;
-2. A Jobtracker running on one of the Resque workers;
-3. One or more Users created in your MongoDB;
-4. One or more Jobs created in a User's Runner;
-
-<a name='section_Start_Start_resque-web'></a>
-### Start resque-web
-
-Mobilize ships with its own rake task to start resque web -- you can do
-the following:
-
-
-  $ MOBILIZE_ENV=<environment> rake mobilize_base:resque_web
-
-This will start a resque_web instance with the port specified in your
-resque.yml and the config/auth scheme specified in your resque_web.rb. 
-
-More detail on the
-[Resque-Web Standalone section][resque-web].
-
-<a name='section_Start_Set_Environment'></a>
-### Set Environment
-
-Mobilize takes the environment from your Rails.env if you're running
-Rails, or assumes "development." You can specify "development", "test",
-or "production," as per the yml files.
-
-Otherwise, it takes it from MOBILIZE_ENV parameter, as in:
-
-``` ruby
-> ENV['MOBILIZE_ENV'] = 'production'
-> require 'mobilize-base'
-```
-This affects all parameters as set in the yml files, including the
-database.
-
-<a name='section_Start_Create_User'></a>
-### Create User
-
-Users are people who use the Mobilize service to move data from one
-endpoint to another. They each have a Runner, which is a google sheet
-that contains one or more Jobs.
-
-To create a requestor, use the User.find_or_create_by_name
-command (replace the user with your own name, or any name
-in your domain).
-
-``` ruby
-irb> User.find_or_create_by_name("user_name")
-```
-
-<a name='section_Start_Start_Workers'></a>
-### Start Workers
-
-Workers are rake tasks that load the Mobilize environment and allow the
-processing of the Jobtracker, Users and Jobs.
-
-These will start as many workers as are defined in your resque.yml.
-
-To start workers, do:
-
-``` ruby
-> Jobtracker.prep_workers
-```
-
-if you have workers already running and would like to kill and refresh
-them, do:
-
-``` ruby
-> Jobtracker.restart_workers!
-```
-
-Note that restart will kill any workers on the Mobilize queue.
-
-<a name='section_Start_View_Logs'></a>
-### View Logs
-
-at this point, you'll want to start viewing the logs for the Resque
-workers -- they will be stored under your log folder, by default log/. You can do:
-
-  $ tail -f log/mobilize-`<environment>`.log
-
-to view them.
-
-<a name='section_Start_Start_Jobtracker'></a>
-### Start Jobtracker
-
-Once the Resque workers are running, and you have at least one User
-set up, it's time to start the Jobtracker:
-
-``` ruby
-> Jobtracker.start
-``` 
-
-The Jobtracker will automatically enqueue any Users that have not
-been processed in the requestor_refresh period defined in the
-jobtracker.yml, and create their Runners if they do not exist. You can
-see this process on your Resque UI and in the log file.
-
-<a name='section_Start_Create_Job'></a>
-### Create Job
-
-Now it's time to go onto the Runner and add a Job to be processed.
-
-To do this, you should log into your Google Drive with either the
-owner's account, an admin account, or the Runner User's account. These
-will be the accounts with edit permissions to a given Runner.
-
-Navigate to the Jobs tab on the Runner `(denoted by Runner(<requestor
-name>))` and enter values under each header:
-
-* name	This is the name of the job you would like to add. Names must be unique across all your jobs, otherwise you will get an error
-	
-* active	set this to blank or FALSE if you want to turn off a job
-	
-* trigger	This uses human readable syntax to schedule jobs. It accepts the following:
-  * every `<integer>` hour --	fire the job at increments of `<integer>` hours, minimum of 1 hour
-  * every `<integer>` day	-- fire the job at increments of `<integer>` days, minimum of 1
-  * every `<integer>` day after <HH:MM>	-- fire the job at increments of <integer> days, after HH:MM UTC time
-  * every `<integer>` day_of_week after <HH:MM>	-- fire the job on specified day of week, after HH:MM UTC time; 1=Sunday
-  * every `<integer>` day_of_month after <HH:MM> -- fire the job on specified day of month, after HH:MM UTC time
-  * once -- fire the job once if active is set to TRUE, set active to FALSE right after
-	* after `<jobname>` -- fire the job after the job named `<jobname>`
-
-* status	Mobilize writes this field with the last status returned by the job
-
-* stage1..stage5 List of stages to be performed by the job. 
-  * Stages have this syntax: `<handler>.<call> <params>`.
-    * handler specifies the file that should receive the stage
-    * the call specifies the method within the file. The method should
-be called `"<handler>.<call>_by_stage_path"`
-    * the params the method accepts, which are custom to each
-stage. These should be of the for `<key1>: <value1>, <key2>: <value2>`, where
-`<key>` is an unquoted string and `<value>` is a quoted string, an
-integer, an array (delimited by square braces), or a hash (delimited by
-curly braces).
-    * For mobilize-base, the following stage is available:
-      * gsheet.write `source: <input_path>`, which reads the sheet. 
-        * The input_path should be of the form:
-          * `<gbook_name>/<gsheet_name>` or just `<gsheet_name>` if the target is in
-the Runner itself. 
-          * `gfile://<gfile_name>` if the target is a file.  
-            * The file must be owned by the Gdrive owner.
-            * The test uses "gfile://test_base_1.tsv".
-    * The stage_name should be of the form `<stage_column>`. The test uses "stage1" for the first test
-and "base1.out" for the second test. The first
-takes the output from the first stage and the second reads it straight
-from the referenced sheet.
-    * All stages accept retry parameters:
-      * retries: an integer specifying the number of times that the system will try it again before giving up.
-      * delay: an integer specifying the number of seconds between retries.
-      * always_on: if false, turns the job off on stage failures.
-Otherwise the job will retry from the beginning with the same frequency as the Runner refresh rate.
-      * notify: by default, the stage owner will be notified on failure.
-          * if false, will not notify the stage owner in the event of a failure. 
-          * If it's an email address, will email the specified person.
-    * If a stage fails after all retries, it will output its standard error to a tab in the Runner with the name of the job, the name of the stage, and a ".err" extension
-      * The tab will be headed "response" and will contain the exception and backtrace for the error.
-    * The test uses "Requestor_mobilize(test)/base1.out" and
-"Runner_mobilize(test)/base2.out" for target sheets.
-
-<a name='section_Start_Run_Test'></a>
-### Run Test
-
-To run tests, you will need to 
-
-1) clone the repository 
-
-From the project folder, run
-
-2) rake mobilize_base:setup
-
-and populate the "test" environment in the config files with the
-necessary details.
-
-3) $ rake test
-
-This will create a test Runner with a sample job. These will run off a
-test redis instance which will be killed once the tests finish.
-
-<a name='section_Start_'></a>
-### Run Test
-
-To run tests, you will need to 
-
-1) clone the repository 
-
-From the project folder, run
-
-2) rake mobilize_base:setup
-
-and populate the "test" environment in the config files with the
-necessary details.
-
-3) $ rake test
-
-This will create a test Runner with a sample job. These will run off a
-test redis instance. This instance will be kept alive so you can test
-additional Mobilize modules. (see [mobilize-ssh][mobilize-ssh] for more)
-
-<a name='section_Start_Add_Gbooks_And_Gsheets'></a>
-### Add Gbooks and Gsheets
-
-A User's Runner should be kept clean, preferably with only the jobs
-sheet. The test keeps everything in the
-Runner, but in reality you will want to create lots of different books
-to share with different people in your organization.
-
-To add a new Gbook, create one as you normally would, then make sure the
-Owner is the same user as specified in your gdrive.yml/owner/name value.
-Mobilize will handle the rest, extending permissions to workers and
-admins.
-
-Also make sure any Gsheets you specify for __read__ operations exist
-prior to calling the job, or there will be an error. __Write__
-operations will create the book and sheet if it does not already exist,
-already under ownership of the owner account.
-
-<a name='section_Meta'></a>
-Meta
-----
-
-* Code: `git clone git://github.com/ngmoco/mobilize-base.git`
-* Home: <https://github.com/ngmoco/mobilize-base>
-* Bugs: <https://github.com/ngmoco/mobilize-base/issues>
-* Gems: <http://rubygems.org/gems/mobilize-base>
-
-<a name='section_Author'></a>
-Author
-------
-
-Cassio Paes-Leme :: cpaesleme@ngmoco.com :: @cpaesleme
-
-<a name='section_Special_Thanks'></a>
-Special Thanks
---------------
-
-* Al Thompson and Sagar Mehta for awesome design advice and discussions
-* Elliott Clark for enlightening me to the wonders of Resque
-* Bob Colner for pointing me to google-drive-ruby when I tried to
-reinvent the wheel
-* ngmoco:) and DeNA Global for supporting and adopting the Mobilize
-platform
-* gimite, defunkt, 10gen, and the countless other github heroes and
-crewmembers.
-
-[google_drive_ruby]: https://github.com/gimite/google-drive-ruby
-[resque]: https://github.com/defunkt/resque
-[mongoid]: http://mongoid.org/en/mongoid/index.html
-[resque_redis]: https://github.com/defunkt/resque#section_Installing_Redis
-[mongodb_quickstart]: http://www.mongodb.org/display/DOCS/Quickstart
-[git_samples]: https://github.com/ngmoco/mobilize-base/tree/master/lib/samples
-[rvm]: https://rvm.io/
-[resque-web]: https://github.com/defunkt/resque#standalone
-[mobilize-ssh]: https://github.com/ngmoco/mobilize-ssh
+Please refer to the mobilize-server wiki: https://github.com/DeNA/mobilize-server/wiki

data/lib/mobilize-base/extensions/array.rb

@@ -12,11 +12,16 @@ class Array
     return self.inject{|sum,x| sum + x }
   end
   def hash_array_to_tsv
-    if self.first.nil? or self.first.class!=Hash
+    ha = self
+    if ha.first.nil? or ha.first.class!=Hash
       return ""
     end
-    header = self.first.keys.join("\t")
-    rows = self.map{|r| r.values.join("\t")}
+    max_row_length = ha.map{|h| h.keys.length}.max
+    header_keys = ha.select{|h| h.keys.length==max_row_length}.first.keys
+    header = header_keys.join("\t")
+    rows = ha.map do |r|
+      header_keys.map{|k| r[k]}.join("\t")
+    end
     ([header] + rows).join("\n")
   end
 end

data/lib/mobilize-base/extensions/google_drive/acl.rb

@@ -14,7 +14,7 @@ module GoogleDrive
     def push(entry)
       #do not send email notifications
       entry = AclEntry.new(entry) if entry.is_a?(Hash)
-      url_suffix = "?send-notification-emails=false"
+      url_suffix = ((@acls_feed_url.index("?") ? "&" : "?") + "send-notification-emails=false")
       header = {"GData-Version" => "3.0", "Content-Type" => "application/atom+xml"}
       doc = @session.request(:post, "#{@acls_feed_url}#{url_suffix}", :data => entry.to_xml(), :header => header, :auth => :writely)
       entry.params = entry_to_params(doc.root)