canvas_sync 0.3.1 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6140a1f5a19c611e6ef92944993bcb0fb676e333
-  data.tar.gz: b26ca4c005bcdaa8fdfd4d6dc4f99342f391601a
+  metadata.gz: 9ef8df28c558a21b5843f195f807b091efcc2cd0
+  data.tar.gz: '0549a9d1691f509116fbc136552b6ed2847ee8e1'
 SHA512:
-  metadata.gz: 2c27000087b00e839a5eb39fec4c03d08c193427ac19cfb8e9757f3103f5167a7f48fdd2f34af75f6257a6a8555d109f92494df2bf89b20e6c75ed6895b79b7b
-  data.tar.gz: 17987763dd6c6e7b47fe1815fa835cd57b4b1f01997879fd8f8279607b3f64b75b5c22eadeaaeeb8e3d0192235edcac69d7972a52e20aebab1a51a6842995700
+  metadata.gz: d4712ea7567083b38968a4256e419546e126466886f1b81898bb018943f01c4471c326f5364d65e390b79ccc8fd26f8240868784847fce9f7748f082e4ed8fe2
+  data.tar.gz: ccd1e2beecb5651f511616ac56848e9282b108188831ca7ee811723ae7d14fadc736fe1a5bc605b7bde9aa3ad82804aaf711a7716718e29a6bc3cae322b0d8f5

data/README.md

@@ -28,15 +28,19 @@ For a list of currently supported models, see `CanvasSync::SUPPORTED_MODELS`.
 
 Additionally, your Canvas instance must have the "Proserv Provisioning Report" enabled.
 
-## Docs
+## Prerequisites
 
-Docs can be generated using [yard](https://yardoc.org/). To view the docs:
+### Postgres
 
-- Clone this gem's repository
-- `bundle install`
-- `yard server --reload`
+The bulk inserting is made possible by using a Postgres upsert. Beause of this, you need to be using **Postgres 9.5** or above.
 
-The yard server will give you a URL you can visit to view the docs.
+### Sidekiq
+
+Make sure you've setup sidekiq to work properly with ActiveJob as [outlined here](https://github.com/mperham/sidekiq/wiki/Active-Job).
+
+### Apartment
+
+If using apartment and sidekiq make sure you include the [apartment-sidekiq](https://github.com/influitive/apartment-sidekiq) gem so that the jobs are run in the correct tenant.
 
 ## Basic Usage
 
@@ -57,7 +61,7 @@ Once that's done and you've used the generator to create your models and migrati
 CanvasSync.provisioning_sync(<array of models to sync>, term_scope: <optional term scope>)
 ```
 
-Note: pass in 'xlist' if you would like sections to include cross listing information
+*Note: pass in 'xlist' to your array of models if you would like sections to include cross listing information*
 
 Example:
 
@@ -69,28 +73,8 @@ This will kick off a string of jobs to sync your specified models.
 
 If you pass in the optional `term_scope` the provisioning reports will be run for only the terms returned by that scope. The scope must be defined on your `Term` model. (A sample one is provided in the generated `Term`.)
 
-Imports are inserted in bulk with [https://github.com/zdennis/activerecord-import](activerecord-import) so they should be very fast.
-
-## Legacy Support
-
-If you have an old style tool that needs to sync data on a row by row basis, you can pass in the `legacy_support: true` option. In order for this to work, your models must have a `create_or_update_from_csv` class method defined that accepts a row argument. This method will get passed each row from the CSV, and it's up to you to persist it.
-
-Example:
-
-```ruby
-CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active, legacy_support: true)
-```
-
-## CanvasSync::JobLog
-
-Running the migrations will create a `canvas_sync_job_logs` table. All the jobs written in this gem will create a `CanvasSync::JobLog` and store data about their arguments, job class, any exceptions, and start/completion time. This will work regardless of your queue adapter.
-
-If you want your own jobs to also log to the table all you have to do is have your job class inherit from `CanvasSync::Job`. You can also persist extra data you might need later by saving to the `metadata` column:
+Imports are inserted in bulk with [activerecord-import](https://github.com/zdennis/activerecord-import) so they should be very fast.
 
-```
-@job_log.metadata = "This job ran really well!"
-@job_log.save!
-```
 
 ## Advanced Usage
 
@@ -125,7 +109,7 @@ class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
       job_chain,
       'my_really_cool_report_csv', # Report name
       { "parameters[param1]" => true }, # Report parameters
-      MyCoolProcessor, # Your processor class
+      MyCoolProcessor.to_s, # Your processor class as a string
       options
     )
   end
@@ -144,7 +128,7 @@ The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to ru
     { job: JobClass, options: {} },
     { job: JobClass2, options: {} }
   ],
-  options: {}
+  global_options: {}
 }
 ```
 
@@ -156,7 +140,7 @@ job_chain = {
     { job: MyReallyCoolReportJob, options: {} },
     { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
   ],
-  options: {}
+  global_options: {}
 }
 
 CanvasSync.process_jobs(job_chain)
@@ -179,7 +163,7 @@ job_chain = {
     { job: SomeRandomJob, options: {} },
     { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
   ],
-  options: {}
+  global_options: {}
 }
 
 CanvasSync.process_jobs(job_chain)
@@ -187,7 +171,46 @@ CanvasSync.process_jobs(job_chain)
 
 ### Batching
 
-The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable if needed.
+The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable.
+
+### Mapping Overrides
+
+Overrides are useful for two scenarios:
+
+- You have an existing application where the column names do not match up with what CanvasSync expects
+- You want to sync some other column in the report that CanvasSync is not configured to sync
+
+In order to create an override, place a file called `canvas_sync_provisioning_mapping.yml` in your Rails `config` directory. Define the tables and columns you want to override using the following format:
+
+```
+users:
+  conflict_target: canvas_user_id # This must be a unique field that is present in the report and the database
+  report_columns: # The keys specified here are the column names in the report CSV
+    canvas_user_id_column_name_in_report:
+        database_column_name: canvas_user_id_name_in_your_db # Sometimes the database column name might not match the report column name
+        type: integer
+```
+
+## Legacy Support
+
+If you have an old style tool that needs to sync data on a row by row basis, you can pass in the `legacy_support: true` option. In order for this to work, your models must have a `create_or_update_from_csv` class method defined that accepts a row argument. This method will get passed each row from the CSV, and it's up to you to persist it.
+
+Example:
+
+```ruby
+CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active, legacy_support: true)
+```
+
+## CanvasSync::JobLog
+
+Running the migrations will create a `canvas_sync_job_logs` table. All the jobs written in this gem will create a `CanvasSync::JobLog` and store data about their arguments, job class, any exceptions, and start/completion time. This will work regardless of your queue adapter.
+
+If you want your own jobs to also log to the table all you have to do is have your job class inherit from `CanvasSync::Job`. You can also persist extra data you might need later by saving to the `metadata` column:
+
+```
+@job_log.metadata = "This job ran really well!"
+@job_log.save!
+```
 
 ## Upgrading
 
@@ -205,28 +228,16 @@ In order for this to work properly your database tables will need to have at lea
 
 When adding to or updating this gem, make sure you do the following:
 
-- If you modify a table that's already in the released version of the gem please write *new migrations* rather than modifying the existing ones. This will allow people to more easily upgrade.
 - Update the yardoc comments where necessary, and confirm the changes by running `yardoc --server`
 - Write specs
 - If you modify the model or migration templates, run `bundle exec rake update_test_schema` to update them in the Rails Dummy application (and commit those changes)
 
+## Docs
 
-## Mapping Overrides
-
-If your local database columns do not match up with those expected by CanvasSync you can define overrides.
-In your Rails config folder, the canvas_sync_provisioning_mapping.yml file contains overrides for custom database column names
-
-```
-users:
-  conflict_target: canvas_user_id # Represents the database column that will determine if we need to update
-  report_columns: # The keys specified here are the column names in the report CSV
-    canvas_user_id_column_name_in_report:
-        database_column_name: canvas_user_id_name_in_your_db
-        type: integer
-```
+Docs can be generated using [yard](https://yardoc.org/). To view the docs:
 
-## TODO
+- Clone this gem's repository
+- `bundle install`
+- `yard server --reload`
 
-- Rethink how options are passed around. The current strategy of having "global_options" and per job options works
-  decently, but can be confusing. The difficulty is representing the options in a way that is easily serializable
-  by the queue adaptor and easily passed around.
+The yard server will give you a URL you can visit to view the docs.

data/lib/canvas_sync/importers/bulk_importer.rb

@@ -8,8 +8,8 @@ module CanvasSync
       # Does a bulk import of a set of models using the activerecord-import gem.
       #
       # @param report_file_path [String] path to the report CSV
-      # @param mapping [Hash] a hash of the values to import. The keys are the CSV column names and
-      #   the values are the database column names. {CanvasSync::Processors::ProvisioningReportProcessor::USERS_CSV_MAPPING Example}
+      # @param mapping [Hash] a hash of the values to import. See `model_mappings.yml` for a
+      #   format example
       # @param klass [Object] e.g., User
       # @param conflict_target [Symbol] represents the database column that will determine if we need to update
       #   or insert a given row. e.g.,: canvas_user_id

data/lib/canvas_sync/version.rb

@@ -1,3 +1,3 @@
 module CanvasSync
-  VERSION = '0.3.1'
+  VERSION = '0.3.2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: canvas_sync
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.3.2
 platform: ruby
 authors:
 - Nate Collings
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-11-17 00:00:00.000000000 Z
+date: 2017-11-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler