mobilize-base 1.0.2 → 1.0.4

data/.gitignore

@@ -7,3 +7,8 @@ lib/mobilize-base/tmp/*
 config/*
 log/*
 .idea/
+config
+log
+tmp
+.DS_Store
+test/dump.rdb

data/LICENSE.txt

@@ -1,20 +1,202 @@
-Copyright (c) Cassio Paes-Leme
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2012 ngmoco, LLC
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+

data/README.md

@@ -11,44 +11,53 @@ Mobilize is an end-to-end data transfer workflow manager with:
 Mobilize-Base includes all the core scheduling and processing
 functionality, allowing you to:
 * put workers on the Mobilize Resque queue.
-* create [Requestors](#section_Start_Requestors_Requestor) and their associated Google Spreadsheet [Jobspecs](#section_Start_Requestors_Jobspec);
-* poll for [Jobs](#section_Job) on Jobspecs (currently gsheet to gsheet only) and add them to Resque;
+* create [Users](#section_Start_Users_User) and their associated Google Spreadsheet [Runners](#section_Start_Users_Runner);
+* poll for [Jobs](#section_Job) on Runners (currently gsheet to gsheet only) and add them to Resque;
 * monitor the status of Jobs on a rolling log.
 
 Table Of Contents
 -----------------
 * [Overview](#section_Overview)
 * [Install](#section_Install)
-    * [Redis](#section_Install_Redis)
-    * [MongoDB](#section_Install_MongoDB)
-    * [Mobilize-Base](#section_Install_Mobilize-Base)
-    * [Default Folders and Files](#section_Install_Folders_and_Files)
+  * [Redis](#section_Install_Redis)
+  * [MongoDB](#section_Install_MongoDB)
+  * [Mobilize-Base](#section_Install_Mobilize-Base)
+  * [Default Folders and Files](#section_Install_Folders_and_Files)
 * [Configure](#section_Configure)
-    * [Google Drive](#section_Configure_Google_Drive)
-    * [Jobtracker](#section_Configure_Jobtracker)
-    * [Mongoid](#section_Configure_Mongoid)
-    * [Resque](#section_Configure_Resque)
+  * [Google Drive](#section_Configure_Google_Drive)
+  * [Jobtracker](#section_Configure_Jobtracker)
+  * [Resque](#section_Configure_Resque)
+  * [Gridfs](#section_Configure_Gridfs)
+  * [Mongoid](#section_Configure_Mongoid)
 * [Start](#section_Start)
-    * [Start resque-web](#section_Start_Start_resque-web)
-    * [Set Environment](#section_Start_Set_Environment)
-    * [Create Requestor](#section_Start_Create_Requestor)
-    * [Start Workers](#section_Start_Start_Workers)
-    * [View Logs](#section_Start_View_Logs)
-    * [Start Jobtracker](#section_Start_Start_Jobtracker)
-    * [Create Job](#section_Start_Create_Job)
-    * [Run Test](#section_Start_Run_Test)
+  * [Start resque-web](#section_Start_Start_resque-web)
+  * [Set Environment](#section_Start_Set_Environment)
+  * [Create User](#section_Start_Create_User)
+  * [Start Workers](#section_Start_Start_Workers)
+  * [View Logs](#section_Start_View_Logs)
+  * [Start Jobtracker](#section_Start_Start_Jobtracker)
+  * [Create Job](#section_Start_Create_Job)
+  * [Run Test](#section_Start_Run_Test)
+  * [Add Gbooks and Gsheets](#section_Start_Add_Gbooks_And_Gsheets)
 * [Meta](#section_Meta)
 * [Author](#section_Author)
+* [Special Thanks](#section_Special_Thanks)
+
 
 <a name='section_Overview'></a>
 Overview
 -----------
 
-* Mobilize is a fun centralized way to access your data lying inside multiple different technoligies under one roof understood by everyone - that is Excel sheets!!
-* Mobilize can enable transfer of data across  diverse databases/technologies like to & from hive, hdfs, hbase, various apis, different databases so that people who are already well versed with dealing with excel sheets can still interact with these diverse technologies and be productive.
-* The spreadsheets are currently hosted in the cloud on Google Spreadsheets, so that you can access them anywhere - even on your tablets.
-* Mobilize in pluggable and extensible, so tomorrow if you want to access data from a cool new database techonology, you can just add a module for that.
-
+* Mobilize is a script deployment and data visualization framework with
+a Google Spreadsheets UI.
+* Mobilize uses Resque for parallelization and queueuing, MongoDB for caching,
+and Google Drive for hosting, user input and display.
+* The [mobilize-ssh][mobilize-ssh] gem allows you to run scripts and
+copy files between different machines, and have output directed to a
+spreadsheet for viewing and processing.
+* The platform is easily extensible: add your own rake tasks and
+handlers by following a few simple conventions, and you can have your own
+Mobilize gem up and running in no time.
 
 <a name='section_Install'></a>
 Install
@@ -70,7 +79,7 @@ instructions.
 ### MongoDB
 
 MongoDB is used to persist caches between reads and writes, keep track
-of Requestors and Jobs, and store Datasets that map to endpoints.
+of Users and Jobs, and store Datasets that map to endpoints.
 
 Please refer to the [MongoDB Quickstart Page][mongodb_quickstart] to get started.
 
@@ -110,21 +119,25 @@ same one that contains your Rakefile)
 Inside the Rakefile in your project's root folder, make sure you have:
 
 ``` ruby
-require 'mobilize-base/tasks'
+require 'mobilize-base/rakes'
 ```
 
-This defines tasks essential to run the environment.
+This defines rake tasks essential to run the environment.
 
 ### Config and Log Folders
 
 run 
 
-  $ rake mobilize:setup
+  $ rake mobilize_base:setup
+
+Mobilize will create config/mobilize/ and log/ folders at the project root
+level. (same as the Rakefile). 
 
-Mobilize will create config and log folders at the project root
-level. (same as the Rakefile)
+(You can override these by passing
+MOBILIZE_CONFIG_DIR and/or MOBILIZE_LOG_DIR arguments to the command.
+All directories must end with a '/'.)
 
-It will also create all required config files, which are detailed below.
+The script will also create samples for all required config files, which are detailed below.
 
 Resque will create a mobilize-resque-`<environment>`.log in the log folder,
 and loop over 10 files, 10MB each.
@@ -133,66 +146,71 @@ and loop over 10 files, 10MB each.
 Configure
 ------------
 
-All Mobilize configurations live in files in `config/*.yml`. Samples can
+All Mobilize configurations live in files in `config/mobilize/*.yml` by
+default. Samples can
 be found below or on github in the [lib/samples][git_samples] folder.
 
 <a name='section_Configure_Google_Drive'></a>
 ### Configure Google Drive
 
-Google drive needs:
-* an owner email address and password. You can set up separate owners
+gdrive.yml needs:
+* a domain, which can be gmail.com but may be different depending on
+your organization. All gdrive accounts should have
+the same domain, and all Users should have emails in this domain.
+* an owner name and password. You can set up separate owners
   for different environments as in the below file, which will keep your
 mission critical workers from getting rate-limit errors.
 * one or more admins with email attributes -- these will be for people
-  who should be given write permissions to ALL Mobilize sheets, for
-maintenance purposes.
-* one or more workers with email and pw attributes -- they will be used
+  who should be given write permissions to all Mobilize books in the
+environment for maintenance purposes.
+* one or more workers with name and pw attributes -- they will be used
   to queue up google reads and writes. This can be the same as the owner
 account for testing purposes or low-volume environments. 
 
 __Mobilize only allows one Resque
 worker at a time to use a Google drive worker account for
-reading/writing.__
+reading/writing, which is called a gdrive_slot.__
 
 Sample gdrive.yml:
 
 ``` yml
-
 development:
+  domain: 'host.com'
   owner:
-    email: 'owner_development@host.com'
+    name: 'owner_development'
     pw: "google_drive_password"
   admins:
-    - {email: 'admin@host.com'}
+    - {name: 'admin'}
   workers:
-    - {email: 'worker_development001@host.com', pw: "worker001_google_drive_password"}
-    - {email: 'worker_development002@host.com', pw: "worker002_google_drive_password"}
+    - {name: 'worker_development001', pw: "worker001_google_drive_password"}
+    - {name: 'worker_development002', pw: "worker002_google_drive_password"}
 test:
+  domain: 'host.com'
   owner:
-    email: 'owner_test@host.com'
+    name: 'owner_test'
     pw: "google_drive_password"
   admins:
-    - {email: 'admin@host.com'}
+    - {name: 'admin'}
   workers:
-    - {email: 'worker_test001@host.com', pw: "worker001_google_drive_password"}
-    - {email: 'worker_test002@host.com', pw: "worker002_google_drive_password"}
+    - {name: 'worker_test001', pw: "worker001_google_drive_password"}
+    - {name: 'worker_test002', pw: "worker002_google_drive_password"}
 production:
+  domain: 'host.com'
   owner:
-    email: 'owner_production@host.com'
+    name: 'owner_production'
     pw: "google_drive_password"
   admins:
-    - {email: 'admin@host.com'}
+    - {name: 'admin'}
   workers:
-    - {email: 'worker_production001@host.com', pw: "worker001_google_drive_password"}
-    - {email: 'worker_production002@host.com', pw: "worker002_google_drive_password"}
-
+    - {name: 'worker_production001', pw: "worker001_google_drive_password"}
+    - {name: 'worker_production002', pw: "worker002_google_drive_password"}
 ```
 
 <a name='section_Configure_Jobtracker'></a>
 ### Configure Jobtracker
 
 The Jobtracker sits on your Resque and does 2 things:
-* check for Requestors that are due for polling;
+* check for Users that are due for polling;
 * send out notifications when:
   * there are failed jobs on Resque;
   * there are jobs on Resque that have run beyond the max run time.
@@ -205,27 +223,83 @@ below and in the [lib/samples][git_samples] folder:
 
 ``` yml
 development:
-  cycle_freq: 10 #10 secs between Jobtracker sweeps
+  cycle_freq: 10 #time between Jobtracker sweeps
   notification_freq: 3600 #1 hour between failure/timeout notifications
-  requestor_refresh_freq: 300 #5 min between requestor checks
+  runner_read_freq: 300 #5 min between runner reads
   max_run_time: 14400 # if a job runs for 4h+, notification will be sent
+  extensions: [] #additional Mobilize modules to load workers with
   admins: #emails to send notifications to
-    - {email: 'admin@host.com'}
+  - {'email': 'admin@host.com'}
 test:
-  cycle_freq: 10 #10 secs between Jobtracker sweeps
+  cycle_freq: 10 #time between Jobtracker sweeps
   notification_freq: 3600 #1 hour between failure/timeout notifications
-  requestor_refresh_freq: 300 #5 min between requestor checks
+  runner_read_freq: 300 #5 min between runner reads
   max_run_time: 14400 # if a job runs for 4h+, notification will be sent
+  extensions: [] #additional Mobilize modules to load workers with
   admins: #emails to send notifications to
-    - {email: 'admin@host.com'}
-
+  - {'email': 'admin@host.com'}
 production:
-  cycle_freq: 10 #10 secs between Jobtracker sweeps
+  cycle_freq: 10 #time between Jobtracker sweeps
   notification_freq: 3600 #1 hour between failure/timeout notifications
-  requestor_refresh_freq: 300 #5 min between requestor checks
+  runner_read_freq: 300 #5 min between runner reads
   max_run_time: 14400 # if a job runs for 4h+, notification will be sent
+  extensions: [] #additional Mobilize modules to load workers with
   admins: #emails to send notifications to
-    - {email: 'admin@host.com'}
+  - {'email': 'admin@host.com'}
+```
+
+<a name='section_Configure_Resque'></a>
+### Configure Resque
+
+Resque keeps track of Jobs, Workers and logging.
+
+It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
+
+* queue_name - the name of the Resque queue where you would like the Jobtracker and Resque Workers to
+  run. Default is mobilize.
+* max_workers - the total number of simultaneous workers you would like
+  on your queue. Default is 4 for development and test, 36 in
+production, but feel free to adjust depending on your hardware.
+* redis_port - you should probably leave this alone, it specifies the
+  default port for dev and prod and a separate one for testing.
+
+``` yml
+development:
+  queue_name: 'mobilize'
+  max_workers: 4
+  redis_port: 6379
+test:
+  queue_name: 'mobilize'
+  max_workers: 4
+  redis_port: 9736
+production:
+  queue_name: 'mobilize'
+  max_workers: 36
+  redis_port: 6379
+```
+
+<a name='section_Configure_Gridfs'></a>
+### Configure Gridfs
+
+Mobilize stores cached data in MongoDB Gridfs. 
+It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
+
+* max_versions - the number of __different__ versions of data to keep
+for a given cache. Default is 10. This is meant mostly to allow you to
+restore Runners from cache if necessary.
+* max_compressed_write_size - the amount of compressed data Gridfs will
+allow. If you try to write more than this, an exception will be thrown.
+
+``` yml
+development:
+  max_versions: 10 #number of versions of cache to keep in gridfs
+  max_compressed_write_size: 1000000000 #~1GB
+test:
+  max_versions: 10 #number of versions of cache to keep in gridfs
+  max_compressed_write_size: 1000000000 #~1GB
+production:
+  max_versions: 10 #number of versions of cache to keep in gridfs
+  max_compressed_write_size: 1000000000 #~1GB
 ```
 
 <a name='section_Configure_Mongoid'></a>
@@ -262,36 +336,6 @@ production:
         - 127.0.0.1:27017
 ```
 
-<a name='section_Configure_Resque'></a>
-### Configure Resque
-
-Resque keeps track of Jobs, Workers and logging.
-
-It needs the below parameters, which can be found in the [lib/samples][git_samples] folder. 
-
-* queue_name - the name of the Resque queue where you would like the Jobtracker and Resque Workers to
-  run. Default is mobilize.
-* max_workers - the total number of simultaneous workers you would like
-  on your queue. Default is 4 for development and test, 36 in
-production, but feel free to adjust depending on your hardware.
-* redis_port - you should probably leave this alone, it specifies the
-  default port for dev and prod and a separate one for testing.
-
-``` yml
-development:
-  queue_name: 'mobilize'
-  max_workers: 4
-  redis_port: 6379
-test:
-  queue_name: 'mobilize'
-  max_workers: 4
-  redis_port: 9736
-production:
-  queue_name: 'mobilize'
-  max_workers: 36
-  redis_port: 6379
-```
-
 <a name='section_Start'></a>
 Start
 -----
@@ -300,8 +344,8 @@ A Mobilize instance can be considered "started" or "running" when you have:
 
 1. Resque workers running on the Mobilize queue;
 2. A Jobtracker running on one of the Resque workers;
-3. One or more Requestors created in your MongoDB;
-4. One or more Jobs created in a Requestor's Jobspec;
+3. One or more Users created in your MongoDB;
+4. One or more Jobs created in a User's Runner;
 
 <a name='section_Start_Start_resque-web'></a>
 ### Start resque-web
@@ -339,26 +383,26 @@ Otherwise, it takes it from MOBILIZE_ENV parameter, set from irb, as in:
 This affects all parameters as set in the yml files, including the
 database.
 
-<a name='section_Start_Create_Requestor'></a>
-### Create Requestor
+<a name='section_Start_Create_User'></a>
+### Create User
 
-Requestors are people who use the Mobilize service to move data from one
-endpoint to another. They each have a Jobspec, which is a google sheet
+Users are people who use the Mobilize service to move data from one
+endpoint to another. They each have a Runner, which is a google sheet
 that contains one or more Jobs.
 
-To create a requestor, use the Requestor.find_or_create_by_email
-command in irb (replace the user with your own email, or any email
-google recognizes).
+To create a requestor, use the User.find_or_create_by_name
+command (replace the user with your own name, or any name
+in your domain).
 
 ``` ruby
-> Requestor.find_or_create_by_email("user@host.com")
+irb> User.find_or_create_by_name("user_name")
 ```
 
 <a name='section_Start_Start_Workers'></a>
 ### Start Workers
 
 Workers are rake tasks that load the Mobilize environment and allow the
-processing of the Jobtracker, Requestors and Jobs.
+processing of the Jobtracker, Users and Jobs.
 
 These will start as many workers as are defined in your resque.yml.
 
@@ -375,13 +419,13 @@ them, do:
 > Jobtracker.restart_workers!
 ```
 
-Note that this will kill any workers on the Mobilize queue.
+Note that restart will kill any workers on the Mobilize queue.
 
 <a name='section_Start_View_Logs'></a>
 ### View Logs
 
 at this point, you'll want to start viewing the logs for the Resque
-workers -- they will be stored under your log folder. You can do:
+workers -- they will be stored under your log folder, by default log/. You can do:
 
   $ tail -f log/mobilize-`<environment>`.log
 
@@ -390,35 +434,35 @@ to view them.
 <a name='section_Start_Start_Jobtracker'></a>
 ### Start Jobtracker
 
-Once the Resque workers are running, and you have at least one Requestor
+Once the Resque workers are running, and you have at least one User
 set up, it's time to start the Jobtracker:
 
 ``` ruby
 > Jobtracker.start
 ``` 
 
-The Jobtracker will automatically enqueue any Requestors that have not
+The Jobtracker will automatically enqueue any Users that have not
 been processed in the requestor_refresh period defined in the
-jobtracker.yml, and create their Jobspecs if they do not exist. You can
+jobtracker.yml, and create their Runners if they do not exist. You can
 see this process on your Resque UI and in the log file.
 
 <a name='section_Start_Create_Job'></a>
 ### Create Job
 
-Now it's time to go onto the Jobspec and add a Job to be processed.
+Now it's time to go onto the Runner and add a Job to be processed.
 
 To do this, you should log into your Google Drive with either the
-owner's account, an admin account, or the Jobspec Requestor's account. These
-will be the accounts with edit permissions to a given Jobspec.
+owner's account, an admin account, or the Runner User's account. These
+will be the accounts with edit permissions to a given Runner.
 
-Navigate to the Jobs tab on the Jobspec `(denoted by Jobspec_<requestor
-name>)` and enter values under each header:
+Navigate to the Jobs tab on the Runner `(denoted by Runner(<requestor
+name>))` and enter values under each header:
 
 * name	This is the name of the job you would like to add. Names must be unique across all your jobs, otherwise you will get an error
 	
 * active	set this to blank or FALSE if you want to turn off a job
 	
-* schedule	This uses human readable syntax to schedule jobs. It accepts the following:
+* trigger	This uses human readable syntax to schedule jobs. It accepts the following:
   * every `<integer>` hour --	fire the job at increments of `<integer>` hours, minimum of 1 hour
   * every `<integer>` day	-- fire the job at increments of `<integer>` days, minimum of 1
   * every `<integer>` day after <HH:MM>	-- fire the job at increments of <integer> days, after HH:MM UTC time
@@ -429,32 +473,47 @@ name>)` and enter values under each header:
 
 * status	Mobilize writes this field with the last status returned by the job
 
-* last_error Mobilize writes any errors to this field, and wipes it if
-  the job completes successfully.
+* task1..task5 List of tasks to be performed by the job. 
+  * Tasks have this syntax: <handler>.<call> <params>.
+    * handler specifies the file that should receive the task
+    * the call specifies the method within the file. The method should
+be called `"<Handler>.<call>_by_task_path"`
+    * the params the method accepts, which are custom to each
+task. These should be a comma-delimited list, with each param in
+quotes.
+    * For mobilize-base, the following tasks are available:
+      * gsheet.read `<input_gsheet_full_path>`, which reads the sheet. 
+        * The gsheet_path should be of the form `<gbook_name>/<gsheet_name>`. The test uses
+"Requestor_mobilize(test)/base1_task1.in".
+      * gsheet.write `<task_relative_path>`,`<output_gsheet_path>`,
+which writes the specified task output to the output_gsheet. 
+        * The task_path should be of the form `<task_column>` or
+`<job_name/task_column>`. The test uses "base1/task1" for the first test
+and simply "task1" for the second test. Both of these take the output
+from the first task.
+        * The test uses "Requestor_mobilize(test)/base1.out" and
+"Requestor_mobilize(test)/base2.out" for output sheets.
 
-* destination_url	Mobilize writes this field with a link to the last dataset returned by the job, blank if none
-	
-* read_handler This is where the job reads its data from. For
-  mobilize-base, you should enter "gsheet"
+<a name='section_Start_Run_Test'></a>
+### Run Test
 
-* write_handler	This is where the job writes its data to. For
-  mobilize-base, you should enter "gsheet"
+To run tests, you will need to 
 
-* param_sheets This is a comma-delimited list of sheets, relayed to the job,
-which can be used for parameters.
-  The format is `<google docs book>/<google docs sheet>`, so if you
-wanted to read from the "output" sheet on the "monthly_results" book you
-would write in `monthly_results/output`. For a sheet in the Jobspec
-itself you could write simply `<output>`.
+1) clone the repository 
 
-* params This is a hash of data, expressed in a JSON, which can be used
-for parameters.
+From the project folder, run
 
-* destination This is the destination for the data, relayed to the job.
-  For a gsheet write_handler, this would be the name of the sheet to be
-written to, similar to param_sheets.
+2) rake mobilize_base:setup
 
-<a name='section_Start_Run_Test'></a>
+and populate the "test" environment in the config files with the
+necessary details.
+
+3) $ rake test
+
+This will create a test Runner with a sample job. These will run off a
+test redis instance which will be killed once the tests finish.
+
+<a name='section_Start_'></a>
 ### Run Test
 
 To run tests, you will need to 
@@ -463,15 +522,34 @@ To run tests, you will need to
 
 From the project folder, run
 
-2) rake mobilize:setup
+2) rake mobilize_base:setup
 
 and populate the "test" environment in the config files with the
 necessary details.
 
 3) $ rake test
 
-This will create a test Jobspec with a sample job. These will run off a
-test redis instance which will be killed once the tests finish.
+This will create a test Runner with a sample job. These will run off a
+test redis instance. This instance will be kept alive so you can test
+additional Mobilize modules. (see [mobilize-ssh][mobilize-ssh] for more)
+
+<a name='section_Start_Add_Gbooks_And_Gsheets'></a>
+### Add Gbooks and Gsheets
+
+A User's Runner should be kept clean, preferably with only the jobs
+sheet. The test keeps everything in the
+Runner, but in reality you will want to create lots of different books
+to share with different people in your organization.
+
+To add a new Gbook, create one as you normally would, then make sure the
+Owner is the same user as specified in your gdrive.yml/owner/name value.
+Mobilize will handle the rest, extending permissions to workers and
+admins.
+
+Also make sure any Gsheets you specify for __read__ operations exist
+prior to calling the job, or there will be an error. __Write__
+operations will create the book and sheet if it does not already exist,
+already under ownership of the owner account.
 
 <a name='section_Meta'></a>
 Meta
@@ -479,7 +557,7 @@ Meta
 
 * Code: `git clone git://github.com/ngmoco/mobilize-base.git`
 * Home: <https://github.com/ngmoco/mobilize-base>
-* Bugs: <https://github.com//mobilize-base/issues>
+* Bugs: <https://github.com/ngmoco/mobilize-base/issues>
 * Gems: <http://rubygems.org/gems/mobilize-base>
 
 <a name='section_Author'></a>
@@ -498,6 +576,8 @@ Special Thanks
 reinvent the wheel
 * ngmoco:) and DeNA Global for supporting and adopting the Mobilize
 platform
+* gimite, defunkt, 10gen, and the countless other github heroes and
+crewmembers.
 
 [google_drive_ruby]: https://github.com/gimite/google-drive-ruby
 [resque]: https://github.com/defunkt/resque
@@ -507,3 +587,4 @@ platform
 [git_samples]: https://github.ngmoco.com/Ngpipes/mobilize-base/tree/master/lib/samples
 [rvm]: https://rvm.io/
 [resque-web]: https://github.com/defunkt/resque#standalone
+[mobilize-ssh]: https://github.com/ngmoco/mobilize-ssh