canvas_sync 0.10.3 → 0.11.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: f220a92d0d85cce6ff10b1ef1a10784039e93583
-  data.tar.gz: 0c305e04af7875261f6c36561b1b39e073f41c41
+SHA256:
+  metadata.gz: bff01b4de68a53faba86076d5390dd4c95d1feec6be899cc28a21ccf2c80a918
+  data.tar.gz: c676786d55e600ed58167c00290cd29e119b62b1bf7974c7ff4df46697759223
 SHA512:
-  metadata.gz: ec6ceaf390ae8a814420d927b1294b66492bd855b62ef9d80042e973871aac7dfb45b061861d503b8ee5303371e67c3df1446be4b159f7f8991ec237684007ad
-  data.tar.gz: 83c67e7bdd93c2ac6e068309955c04a08e8f6204481f667049cb3ac6dc8ba5212c31f5e25febde46a788bc5582d18c893cf421718f7604a2399623289e9ff422
+  metadata.gz: c7340c9158ecc077e4f2246c24c6b4a608229cc2568a74b25b2afe8d7bebfd42e79deb23b49d22a7d7842e9a7923caaa1a43843fc68fa04497901477d480e9a2
+  data.tar.gz: a4f9ca68735ccc0ceb42cc9f8b486dd18b02e508b7ece0242b1a56c49c33db7e426126f0cb3d7a5bd5ba1057ee69082fe66ecfad252cab0fcb5d16c9c3fc12f4

data/README.md

@@ -91,6 +91,29 @@ This gem also helps with syncing and processing other reports if needed. In orde
 - Integrate your reports with the `ReportStarter`
 - Tell the gem what jobs to run
 
+### Extensible chain
+It is sometimes desired to extend or customize the chain of jobs that are run with CanvasSync.
+This can be achieved with the following pattern:
+
+```ruby
+job_chain = CanvasSync.default_provisioning_report_chain(
+  %i[list of models to sync]
+)
+
+# CanvasSync forks the chain for each term within a term scope. The ForkGather Job can be used to unfork the chain.
+# For example multiple Terms are in the term_scope. CanvasSync syncs Accounts, Terms, and Users (if enabled) and then
+# forks the chain to sync other models (eg Course, Sections, etc.) per-Term.
+# ForkGather will wait until all the forked chains are complete before continuing.
+# TL;DR: Jobs placed after SyncProvisioningReportJob and before ForkGather will run once per Term per Sync;
+# Jobs placed before SyncProvisioningReportJob or after ForkGather will run once per Sync
+job_chain[:jobs] << { job: CanvasSync::Jobs::ForkGather, options: {} }
+
+# Add a custom job to the end of the chain. Custom jobs must accept 2 arguments (job_chain, options) and call CanvasSync.invoke_next(job_chain) when complete
+job_chain[:jobs] << { job: CanvasSyncCompleteWorker, options: { job_id: job.id } }
+
+CanvasSync.invoke_next(job_chain)
+```
+
 ### Processor
 
 Your processor class must implement a `process` class method that receives a `report_file_path` and a hash of `options`. (See the `CanvasSync::Processors::ProvisioningReportProcessor` for an example.) The gem handles the work of enqueueing and downloading the report and then passes the file path to your class to process as needed. A simple example might be:
@@ -141,7 +164,7 @@ The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to ru
 
 Here is an example that runs our new report job first followed by the builtin provisioning job:
 
-```
+```ruby
 job_chain = {
   jobs: [
     { job: MyReallyCoolReportJob, options: {} },
@@ -156,7 +179,7 @@ CanvasSync.process_jobs(job_chain)
 What if you've got some other job that you want run that doesn't deal with a report? No problem! Just make sure you call `CanvasSync.invoke_next` at the end of your job. Example:
 
 
-```
+```ruby
 class SomeRandomJob < CanvasSync::Job
   def perform(job_chain, options)
     i_dunno_do_something!
@@ -375,6 +398,38 @@ Available config options (if you add more, please update this!):
 
 * `config.classes_to_only_log_errors_on` - use this if you are utilizing the `CanvasSync::JobLog` table, but want certain classes to only persist in the `job_logs` table if an error is encountered. This is useful if you've got a very frequently used job that's filling up your database, and only really care about tracking failures.
 
+## Handling Job errors
+
+If you need custom handling for when a CanvasSync Job fails, you can add an `:on_failure` option to you Job Chain's `:global_options`.
+The value should be a String in the following format: `ModuleOrClass::AnotherModuleOrClass.class_method`.
+The given method of the given class will be called when an error occurs.
+The handling method should accept 2 arguments: `[error, **options]`
+
+The current parameters provided in `**options` are:
+- `job_chain`
+- `job_log`
+
+Example:
+```ruby
+class CanvasSyncStarterWorker
+  def perform
+    job_chain = CanvasSync.default_provisioning_report_chain(
+      %w[desired models],
+      options: {
+        global: {
+          on_failure: 'CanvasSyncStarterWorker.handle_canvas_sync_error',
+        }
+      }
+    )
+    CanvasSync.invoke_next(job_chain)
+  end
+
+  def self.handle_canvas_sync_error(error, **options)
+    # Do Stuff
+  end
+end
+```
+
 ## Upgrading
 
 Re-running the generator when there's been a gem change will give you several choices if it detects conflicts between your local files and the updated generators. You can either view a diff or allow the generator to overwrite your local file. In most cases you may just want to add the code from the diff yourself so as not to break any of your customizations.

data/db/migrate/20190916154829_add_fork_count_to_canvas_sync_job_logs.rb

@@ -0,0 +1,13 @@
+if Rails.version.to_f >= 5.0
+  class AddForkCountToCanvasSyncJobLogs < ActiveRecord::Migration[Rails.version.to_f]
+    def change
+      add_column :canvas_sync_job_logs, :fork_count, :integer
+    end
+  end
+else
+  class AddForkCountToCanvasSyncJobLogs < ActiveRecord::Migration
+    def change
+      add_column :canvas_sync_job_logs, :fork_count, :integer
+    end
+  end
+end

data/lib/canvas_sync.rb

@@ -6,6 +6,7 @@ require "canvas_sync/sidekiq_job"
 require "canvas_sync/api_syncable"
 require "canvas_sync/jobs/report_starter"
 require "canvas_sync/jobs/report_checker"
+require "canvas_sync/jobs/fork_gather"
 require "canvas_sync/jobs/report_processor_job"
 require "canvas_sync/jobs/sync_provisioning_report_job"
 require "canvas_sync/jobs/sync_simple_table_job.rb"
@@ -100,12 +101,16 @@ module CanvasSync
       invoke_next(job_chain)
     end
 
+    def duplicate_chain(job_chain)
+      Marshal.load(Marshal.dump(job_chain))
+    end
+
     # Invokes the next job in a chain of jobs.
     #
     # This should typically be called automatically by the gem where necessary.
     #
     # @param job_chain [Hash] A chain of jobs to execute
-    def invoke_next(job_chain)
+    def invoke_next(job_chain, extra_options: {})
       return if job_chain[:jobs].empty?
 
       # Make sure all job classes are serialized as strings
@@ -115,7 +120,32 @@ module CanvasSync
       jobs = duped_job_chain[:jobs]
       next_job = jobs.shift
       next_job_class = next_job[:job].constantize
-      next_job_class.perform_later(duped_job_chain, next_job[:options])
+      next_options = next_job[:options] || {}
+      next_options.merge!(extra_options)
+      next_job_class.perform_later(duped_job_chain, next_options)
+    end
+
+    def fork(job_log, job_chain, keys: [])
+      duped_job_chain = Marshal.load(Marshal.dump(job_chain))
+      duped_job_chain[:global_options][:fork_path] ||= []
+      duped_job_chain[:global_options][:fork_keys] ||= []
+      duped_job_chain[:global_options][:fork_path] << job_log.job_id
+      duped_job_chain[:global_options][:fork_keys] << ['canvas_term_id']
+      duped_job_chain[:global_options][:on_failure] ||= 'CanvasSync::Jobs::ForkGather.handle_branch_error'
+      sub_items = yield duped_job_chain
+      sub_count = sub_items.respond_to?(:count) ? sub_items.count : sub_items
+      job_log.fork_count = sub_count
+      sub_items
+    end
+
+    # Given a Model or Relation, scope it down to items that should be synced
+    def sync_scope(scope)
+      terms = %i[should_canvas_sync active_for_canvas_sync should_sync active_for_sync active]
+      terms.each do |t|
+        return scope.send(t) if scope.respond_to?(t)
+      end
+      Rails.logger.warn("Could not filter Syncable Scope for model '#{scope.try(:model)&.name || scope.name}'")
+      scope
     end
 
     # Syn any report to an specific set of models
@@ -159,7 +189,7 @@ module CanvasSync
       return unless models.present?
       models.map! &:to_s
       term_scope = term_scope.to_s if term_scope
-      options = options.with_indifferent_access
+      options = options.deep_symbolize_keys!
 
       model_job_map = {
         terms: CanvasSync::Jobs::SyncTermsJob,
@@ -177,7 +207,7 @@ module CanvasSync
       jobs = []
       try_add_model_job = ->(model) {
         return unless models.include?(model)
-        jobs.push(job: model_job_map[model].to_s, options: options[model] || {})
+        jobs.push(job: model_job_map[model].to_s, options: options[model.to_sym] || {})
         models -= [model]
       }
 
@@ -210,14 +240,19 @@ module CanvasSync
       post_provisioning_jobs = jobs
 
       jobs = pre_provisioning_jobs
-      jobs += Array.wrap({
-        job: CanvasSync::Jobs::SyncProvisioningReportJob.to_s,
-        options: { term_scope: term_scope, models: models },
-      }) if models.present?
+      if models.present?
+        provisioning_job = {
+          job: CanvasSync::Jobs::SyncProvisioningReportJob.to_s,
+          options: { term_scope: term_scope, models: models },
+        }
+        provisioning_job[:options].merge!(options[:provisioning]) if options[:provisioning].present?
+        jobs += Array.wrap(provisioning_job)
+      end
       jobs += post_provisioning_jobs
 
       global_options = { legacy_support: legacy_support }
       global_options[:account_id] = account_id if account_id.present?
+      global_options.merge!(options[:global]) if options[:global].present?
 
       { jobs: jobs, global_options: global_options }
     end

data/lib/canvas_sync/api_syncable.rb

@@ -21,6 +21,14 @@ module CanvasSync::ApiSyncable
       inst
     end
 
+    def create_or_update_from_api_params(api_params)
+      api_params = api_params.with_indifferent_access
+      inst = find_or_initialize_by(canvas_id: api_params[:id])
+      inst.update_from_api_params(api_params)
+      inst.save! if inst.changed?
+      inst
+    end
+
     def api_sync_options=(opts)
       @api_sync_options = opts
     end

data/lib/canvas_sync/generators/templates/migrations/create_admins.rb

@@ -5,13 +5,16 @@ class CreateAdmins < ActiveRecord::Migration[5.1]
     create_table :admins do |t|
       t.bigint :canvas_id, null: false
       t.string :role_name
-      t.bigint :canvas_role_id, null: false
-      t.json :user_data
-      t.bigint :canvas_user_id, null: false
-      t.string :workflow_state, null: false
+      t.bigint :canvas_account_id
+      t.bigint :canvas_role_id
+      t.bigint :canvas_user_id
+      t.string :workflow_state
 
       t.timestamps
     end
     add_index :admins, :canvas_id, unique: true
+    add_index :admins, :canvas_role_id
+    add_index :admins, :canvas_user_id
+    add_index :admins, :canvas_account_id
   end
 end

data/lib/canvas_sync/generators/templates/migrations/create_roles.rb

@@ -4,11 +4,10 @@ class CreateRoles < ActiveRecord::Migration[5.1]
   def change
     create_table :roles do |t|
       t.integer :canvas_id, null: false
-      t.string :label, null: false
-      t.string :base_role_type, null: false
-      t.json :account
+      t.string :label
+      t.string :base_role_type
       t.integer :canvas_account_id
-      t.string :workflow_state, null: false
+      t.string :workflow_state
       t.json :permissions
 
       t.timestamps

data/lib/canvas_sync/generators/templates/models/account.rb

@@ -5,6 +5,8 @@ class Account < ApplicationRecord
 
   validates :canvas_id, uniqueness: true, presence: true
 
+  has_many :admins, primary_key: :canvas_id, foreign_key: :canvas_account_id
+
   api_syncable({
     name: :name,
     workflow_state: :workflow_state,

data/lib/canvas_sync/generators/templates/models/admin.rb

@@ -4,6 +4,7 @@ class Admin < ApplicationRecord
   include CanvasSync::ApiSyncable
 
   validates :canvas_id, uniqueness: true, presence: true
+  belongs_to :account, primary_key: :canvas_id, foreign_key: :canvas_account_id, optional: true
   belongs_to :user, primary_key: :canvas_id, foreign_key: :canvas_user_id, optional: true
   belongs_to :role, primary_key: :canvas_id, foreign_key: :canvas_role_id, optional: true
 
@@ -11,16 +12,14 @@ class Admin < ApplicationRecord
     canvas_id: :id,
     role_name: :role,
     canvas_role_id: :role_id,
-    user_data: :user,
     canvas_user_id: ->(r) { r['user']['id'] },
+    # NOTICE: The :account_id field is added by CanvasSync - it is not included in the Canvas API response
+    canvas_account_id: :account_id,
     workflow_state: :workflow_state,
   }, -> (api) {
-    admins = api.account_admins('self').all_pages!
+    admins = api.account_admins(canvas_account_id).all_pages!
     admin_data = admins.find{|admin| admin['id'] == canvas_id }
     raise Footrest::HttpError::NotFound unless admin_data.present?
     admin_data
   })
-
-  def self.create_or_update(params); find_or_initialize_by(canvas_id: params['id']).update_from_api_params!(params); end
-
 end

data/lib/canvas_sync/generators/templates/models/role.rb

@@ -10,12 +10,8 @@ class Role < ApplicationRecord
     canvas_id: :id,
     label: :label,
     base_role_type: :base_role_type,
-    account: :account,
     canvas_account_id: ->(r) { r.dig('account', 'id') },
     permissions: :permissions,
     workflow_state: :workflow_state,
   }, -> (api) { api.get("/api/v1/accounts/#{canvas_account_id}/roles/#{canvas_id}") })
-
-  def self.create_or_update(params); find_or_initialize_by(canvas_id: params['id']).update_from_api_params!(params); end
-
 end

data/lib/canvas_sync/generators/templates/models/term.rb

@@ -21,8 +21,6 @@ class Term < ApplicationRecord
     term_data
   })
 
-  def self.create_or_update(params); find_or_initialize_by(canvas_id: params['id']).update_from_api_params!(params); end
-
   # This is a sample scope created by the CanvasSync gem; feel
   # free to customize it for your tool's requirements.
   scope :active, -> {

data/lib/canvas_sync/importers/bulk_importer.rb

@@ -58,13 +58,12 @@ module CanvasSync
         columns = columns.dup
 
         update_conditions = {
-          condition: condition_sql(klass, columns),
+          condition: condition_sql(klass, columns, import_args[:sync_start_time]),
           columns: columns
         }
         update_conditions[:conflict_target] = conflict_target if conflict_target
 
         options = { validate: false, on_duplicate_key_update: update_conditions }.merge(import_args)
-
         options.delete(:on_duplicate_key_update) if options.key?(:on_duplicate_key_ignore)
         klass.import(columns, rows, options)
       end
@@ -79,10 +78,14 @@ module CanvasSync
       # started_at = Time.now
       # run_the_users_sync!
       # changed = User.where("updated_at >= ?", started_at)
-      def self.condition_sql(klass, columns)
+      def self.condition_sql(klass, columns, report_start)
         columns_str = columns.map { |c| "#{klass.quoted_table_name}.#{c}" }.join(", ")
         excluded_str = columns.map { |c| "EXCLUDED.#{c}" }.join(", ")
-        "(#{columns_str}) IS DISTINCT FROM (#{excluded_str})"
+        condition_sql = "(#{columns_str}) IS DISTINCT FROM (#{excluded_str})"
+        if klass.column_names.include?("updated_at") && report_start
+          condition_sql += " AND #{klass.quoted_table_name}.updated_at < '#{report_start}'"
+        end
+        condition_sql
       end
 
       def self.batch_size

data/lib/canvas_sync/job.rb

@@ -13,13 +13,26 @@ module CanvasSync
       @job_log.started_at = Time.now
       @job_log.save
 
+      @job_chain = job.arguments[0] if job.arguments[0].is_a?(Hash) && job.arguments[0].include?(:jobs)
+
       begin
         block.call
         @job_log.status = JobLog::SUCCESS_STATUS
       rescue => e # rubocop:disable Style/RescueStandardError
         @job_log.exception = "#{e.class}: #{e.message}"
-        @job_log.backtrace = e.backtrace
+        @job_log.backtrace = e.backtrace.join('\n')
         @job_log.status = JobLog::ERROR_STATUS
+        if @job_chain&.[](:global_options)&.[](:on_failure)&.present?
+          begin
+            class_name, method = @job_chain[:global_options][:on_failure].split('.')
+            klass = class_name.constantize
+            klass.send(method.to_sym, e, job_chain: @job_chain, job_log: @job_log)
+          rescue => e2
+            @job_log.backtrace += "\n\nError Occurred while handling an Error: #{e2.class}: #{e2.message}"
+            @job_log.backtrace += "\n" + e2.backtrace.join('\n')
+            Raven.captureException(e2) if defined? Raven
+          end
+        end
         raise e
       ensure
         if CanvasSync.config.classes_to_only_log_errors_on.include?(@job_log.job_class) && @job_log.status != JobLog::ERROR_STATUS
@@ -43,5 +56,15 @@ module CanvasSync
         status: JobLog::ENQUEUED_STATUS,
       )
     end
+
+    def update_or_create_model(model, params)
+      if model.respond_to? :create_or_update
+        model.create_or_update(params)
+      elsif model.method_defined? :update_from_api_params!
+        model.find_or_initialize_by(canvas_id: params['id']).update_from_api_params!(params)
+      else
+        raise "Could not create/update #{model.name}. It must have a create_or_update(params) ClassMethod or implement ApiSyncable."
+      end
+    end
   end
 end

data/lib/canvas_sync/jobs/fork_gather.rb

@@ -0,0 +1,59 @@
+module CanvasSync
+  module Jobs
+    class ForkGather < CanvasSync::Job
+      def perform(job_chain, options)
+        forked_job = self.class.forked_at_job(job_chain)
+
+        if forked_job.present?
+          forked_job.with_lock do
+            forked_job.fork_count -= 1
+            forked_job.save!
+          end
+
+          if forked_job.fork_count <= 0
+            (job_chain[:global_options][:fork_keys] || []).pop&.each do |k|
+              job_chain[:global_options].delete(k.to_sym)
+            end
+            CanvasSync.invoke_next(job_chain)
+          end
+        else
+          CanvasSync.invoke_next(job_chain)
+        end
+      end
+
+      def self.handle_branch_error(e, job_chain:, skip_invoke: false, **kwargs)
+        return nil unless job_chain&.[](:global_options)&.[](:fork_path).present?
+
+        duped_chain = CanvasSync.duplicate_chain(job_chain)
+        job_list = duped_chain[:jobs]
+        while job_list.count > 0
+          job_class_name = job_list[0][:job]
+          job_class = job_class_name.constantize
+          break if job_class <= CanvasSync::Jobs::ForkGather
+          job_list.shift
+        end
+
+        return nil unless job_list.present?
+
+        if skip_invoke
+          duped_chain
+        else
+          CanvasSync.invoke_next(duped_chain)
+          true
+        end
+      end
+
+      protected
+
+      def self.forked_at_job(job_chain)
+        fork_item = (job_chain[:global_options][:fork_path] || []).pop
+
+        if fork_item.present?
+          CanvasSync::JobLog.find_by(job_id: fork_item)
+        else
+          nil
+        end
+      end
+    end
+  end
+end