canvas_sync 0.16.4 → 0.17.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ddf6a257acf95c92ba99a6d03c9bcdd99ba1a8ff
-  data.tar.gz: f10e8f91a225462545354a24daaf006a170bd90d
+  metadata.gz: b4ffe8d3a42b647f8c30f1d831c4a86998bc7c1b
+  data.tar.gz: 2ba762a479d2df35e22a69b7930d051a2e43bd30
 SHA512:
-  metadata.gz: cde3a3a534316a0faa535347a6c71aa63ab46c5cc5d9c21b01512bb623bed7f6b34fc7da38de345fefcf238cc44f0423f849dd08aa8f57f292472838aa6f7d0c
-  data.tar.gz: 2e1b8217af038166f60afe4032f0f1ec806761fb81d5727aec57540919146335f7a28292e8e81ba87220eb2d687325b8145be798ed6a29b07aaad0cd6e3b56f4
+  metadata.gz: 6702f6754b217edfe2c7e4ff51c586940e7073993b5e3a79464e49308edd274af9bc8165f38aa919ef3c288ec86f1623065a91f42121f68c4da1389f27a57cb2
+  data.tar.gz: 71365002bcf14b762b00b73ae805862d8691a2b78ecd0254c0ef7fcf56eea6caeae838652e7ca4a685834eb380a08e782cf78e591b5af2c8950b4d4fd763e704

data/README.md

@@ -91,27 +91,45 @@ This gem also helps with syncing and processing other reports if needed. In orde
 - Integrate your reports with the `ReportStarter`
 - Tell the gem what jobs to run
 
+### `updated_after`
+An `updated_after` param may be passed when triggering a provision or making a chain:
+```ruby
+CanvasSync.default_provisioning_report_chain(
+  %i[list of models to sync], updated_after: false
+)
+```
+It may be one of the following values:
+* `false` - Will not apply any `updated_after` filtering to the requested reports
+* An ISO-8601 Date - Will pass the supplied date ad the `updated_after` param for the requested reports
+* `true` (Default) - Will use the start date of the last successful sync
+
 ### Extensible chain
 It is sometimes desired to extend or customize the chain of jobs that are run with CanvasSync.
 This can be achieved with the following pattern:
 
 ```ruby
-job_chain = CanvasSync.default_provisioning_report_chain(
+chain = CanvasSync.default_provisioning_report_chain(
   %i[list of models to sync]
 )
 
-# CanvasSync forks the chain for each term within a term scope. The ForkGather Job can be used to unfork the chain.
-# For example multiple Terms are in the term_scope. CanvasSync syncs Accounts, Terms, and Users (if enabled) and then
-# forks the chain to sync other models (eg Course, Sections, etc.) per-Term.
-# ForkGather will wait until all the forked chains are complete before continuing.
-# TL;DR: Jobs placed after SyncProvisioningReportJob and before ForkGather will run once per Term per Sync;
-# Jobs placed before SyncProvisioningReportJob or after ForkGather will run once per Sync
-job_chain[:jobs] << { job: CanvasSync::Jobs::ForkGather, options: {} }
+# Add a custom job to the end of the chain.
+chain << { job: CanvasSyncCompleteWorker, parameters: [{ job_id: job.id }] }
+chain << { job: CanvasSyncCompleteWorker, options: { job_id: job.id } } # If an options key is provided, it will be automatically appended to the end of the :parameters array
+
+chain.process!
 
-# Add a custom job to the end of the chain. Custom jobs must accept 2 arguments (job_chain, options) and call CanvasSync.invoke_next(job_chain) when complete
-job_chain[:jobs] << { job: CanvasSyncCompleteWorker, options: { job_id: job.id } }
+# The chain object provides a fairly extensive API:
+chain.insert({ job: SomeOtherJob }) # Adds the job to the end of the chain
+chain.insert_at(0, { job: SomeOtherJob }) # Adds the job to the beginning of the chain
+chain.insert({ job: SomeOtherJob }, after: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job right after the SyncTermsJob
+chain.insert({ job: SomeOtherJob }, before: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job right before the SyncTermsJob
+chain.insert({ job: SomeOtherJob }, with: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job to be performed concurrently with the SyncTermsJob
 
-CanvasSync.invoke_next(job_chain)
+# Some Jobs (such as the SyncTermsJob) have a sub-chain for, eg, Courses.
+# chain.insert is aware of these sub-chains and will recurse into them when looking for a before:/after:/with: reference
+chain.insert({ job: SomeOtherJob }, after: 'CanvasSync::Jobs::SyncCoursesJob') # Adds the job to be performed after SyncCoursesJob (which is a sub-job of the terms job and is duplicated for each term in the term_scope:)
+# You can also retrieve the sub-chain like so:
+chain.get_sub_chain('CanvasSync::Jobs::SyncTermsJob')
 ```
 
 ### Processor
@@ -134,9 +152,8 @@ Let's say we have a custom Canvas report called "my_really_cool_report_csv". Fir
 
 ```ruby
 class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
-  def perform(job_chain, options)
+  def perform(options)
     super(
-      job_chain,
       'my_really_cool_report_csv', # Report name
       { "parameters[param1]" => true }, # Report parameters
       MyCoolProcessor.to_s, # Your processor class as a string
@@ -148,57 +165,6 @@ end
 
 You can also see examples in `lib/canvas_sync/jobs/sync_users_job.rb` and `lib/canvas_sync/jobs/sync_provisioning_report.rb`.
 
-### Start the jobs
-
-The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to run. The job chain must be formatted like:
-
-```ruby
-{
-  jobs: [
-    { job: JobClass, options: {} },
-    { job: JobClass2, options: {} }
-  ],
-  global_options: {}
-}
-```
-
-Here is an example that runs our new report job first followed by the builtin provisioning job:
-
-```ruby
-job_chain = {
-  jobs: [
-    { job: MyReallyCoolReportJob, options: {} },
-    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
-  ],
-  global_options: {}
-}
-
-CanvasSync.process_jobs(job_chain)
-```
-
-What if you've got some other job that you want run that doesn't deal with a report? No problem! Just make sure you call `CanvasSync.invoke_next` at the end of your job. Example:
-
-
-```ruby
-class SomeRandomJob < CanvasSync::Job
-  def perform(job_chain, options)
-    i_dunno_do_something!
-
-    CanvasSync.invoke_next(job_chain)
-  end
-end
-
-job_chain = {
-  jobs: [
-    { job: SomeRandomJob, options: {} },
-    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
-  ],
-  global_options: {}
-}
-
-CanvasSync.process_jobs(job_chain)
-```
-
 ### Batching
 
 The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable.
@@ -257,6 +223,14 @@ class CanvasSyncModel < ApplicationRecord
 end
 ```
 
+### Job Batching
+CanvasSync adds a `CanvasSync::JobBatches` module. It adds Sidekiq/sidekiq-batch like support for Job Batches.
+It integrates automatically with both Sidekiq and ActiveJob. The API is highly similar to the Sidekiq-batch implementation,
+documentation for which can be found at https://github.com/mperham/sidekiq/wiki/Batches
+
+A batch can be created using `Sidekiq::Batch` or `CanvasSync::JobBatching::Batch`.
+
+Also see `canvas_sync/jobs/begin_sync_chain_job`, `canvas_sync/Job_batches/jobs/serial_batch_job`, or `canvas_sync/Job_batches/jobs/concurrent_batch_job` for example usage.
 
 ## Legacy Support
 
@@ -308,80 +282,19 @@ end
 ## Syncronize different reports
 CanvasSync provides the functionality to import data from other reports into an specific table.
 
-This can be achived by using the followin method
-
-```ruby
-CanvasSync.provisioning_sync(<array of models to sync>, term_scope: <optional term scope>)
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: <report name>,
-        model: <model to sync>,
-        params: <hash with the require parameters the report needs to sync>
-      },
-      {
-        report_name: <report name>,
-        model: <model to sync>,
-        params: <hash with the require parameters the report needs to sync>
-      },
-      ...
-    ],
-    term_scope: <optional term scope>
-  )
-```
-
-Example:
+This can be achieved by using the following method
 
 ```ruby
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'users',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[users]" => true
-        }
-      },
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'accounts',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[accounts]" => true
-        }
-      }
-    ]
-  )
-```
-
-Example with the term_scope active:
-
-```ruby
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'sections',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[sections]" => true
-        }
-      },
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'courses',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[courses]" => true
-        }
-      }
-    ],
-    term_scope: 'active'
-  )
+chain = CanvasSync.default_provisioning_report_chain
+chain << {
+  job: CanvasSync::Jobs::SyncSimpleTableJob,
+  options: {
+    report_name: <report name>,
+    model: <model to sync>,
+    params: <hash with the require parameters the report needs to sync>
+  },
+}
+chain.process!
 ```
 
 ## Configuration
@@ -421,7 +334,6 @@ class CanvasSyncStarterWorker
         }
       }
     )
-    CanvasSync.invoke_next(job_chain)
   end
 
   def self.handle_canvas_sync_error(error, **options)

data/app/models/canvas_sync/sync_batch.rb

@@ -0,0 +1,5 @@
+module CanvasSync
+  class SyncBatch < ApplicationRecord
+    serialize :job_arguments, Array
+  end
+end

data/db/migrate/20170915210836_create_canvas_sync_job_log.rb

@@ -1,35 +1,16 @@
-if Rails.version.to_f >= 5.0
-  class CreateCanvasSyncJobLog < ActiveRecord::Migration[Rails.version.to_f]
-    def change
-      create_table :canvas_sync_job_logs do |t|
-        t.datetime :started_at
-        t.datetime :completed_at
-        t.string :exception
-        t.text :backtrace
-        t.string :job_class
-        t.string :status
-        t.text :metadata
-        t.text :job_arguments
+class CreateCanvasSyncJobLog < CanvasSync::MiscHelper::MigrationClass
+  def change
+    create_table :canvas_sync_job_logs do |t|
+      t.datetime :started_at
+      t.datetime :completed_at
+      t.string :exception
+      t.text :backtrace
+      t.string :job_class
+      t.string :status
+      t.text :metadata
+      t.text :job_arguments
 
-        t.timestamps
-      end
-    end
-  end
-else
-  class CreateCanvasSyncJobLog < ActiveRecord::Migration
-    def change
-      create_table :canvas_sync_job_logs do |t|
-        t.datetime :started_at
-        t.datetime :completed_at
-        t.string :exception
-        t.text :backtrace
-        t.string :job_class
-        t.string :status
-        t.text :metadata
-        t.text :job_arguments
-
-        t.timestamps
-      end
+      t.timestamps
     end
   end
 end

data/db/migrate/20180725155729_add_job_id_to_canvas_sync_job_logs.rb

@@ -1,15 +1,6 @@
-if Rails.version.to_f >= 5.0
-  class AddJobIdToCanvasSyncJobLogs < ActiveRecord::Migration[Rails.version.to_f]
-    def change
-      add_column :canvas_sync_job_logs, :job_id, :string
-      add_index :canvas_sync_job_logs, :job_id
-    end
-  end
-else
-  class AddJobIdToCanvasSyncJobLogs < ActiveRecord::Migration
-    def change
-      add_column :canvas_sync_job_logs, :job_id, :string
-      add_index :canvas_sync_job_logs, :job_id
-    end
+class AddJobIdToCanvasSyncJobLogs < CanvasSync::MiscHelper::MigrationClass
+  def change
+    add_column :canvas_sync_job_logs, :job_id, :string
+    add_index :canvas_sync_job_logs, :job_id
   end
 end

data/db/migrate/20190916154829_add_fork_count_to_canvas_sync_job_logs.rb

@@ -1,13 +1,5 @@
-if Rails.version.to_f >= 5.0
-  class AddForkCountToCanvasSyncJobLogs < ActiveRecord::Migration[Rails.version.to_f]
-    def change
-      add_column :canvas_sync_job_logs, :fork_count, :integer
-    end
-  end
-else
-  class AddForkCountToCanvasSyncJobLogs < ActiveRecord::Migration
-    def change
-      add_column :canvas_sync_job_logs, :fork_count, :integer
-    end
+class AddForkCountToCanvasSyncJobLogs < CanvasSync::MiscHelper::MigrationClass
+  def change
+    add_column :canvas_sync_job_logs, :fork_count, :integer
   end
 end

data/db/migrate/20201018210836_create_canvas_sync_sync_batches.rb

@@ -0,0 +1,11 @@
+class CreateCanvasSyncSyncBatches < CanvasSync::MiscHelper::MigrationClass
+  def change
+    create_table :canvas_sync_sync_batches do |t|
+      t.datetime :started_at
+      t.datetime :completed_at
+      t.string :status
+
+      t.timestamps
+    end
+  end
+end

data/lib/canvas_sync.rb

@@ -2,9 +2,9 @@ require "bearcat"
 
 require "canvas_sync/version"
 require "canvas_sync/engine"
+require "canvas_sync/misc_helper"
 require "canvas_sync/class_callback_executor"
 require "canvas_sync/job"
-require "canvas_sync/job_chain"
 require "canvas_sync/sidekiq_job"
 require "canvas_sync/api_syncable"
 require "canvas_sync/record"
@@ -13,6 +13,8 @@ require "canvas_sync/jobs/report_checker"
 require "canvas_sync/jobs/report_processor_job"
 require "canvas_sync/config"
 
+require "canvas_sync/job_batches/batch"
+
 Dir[File.dirname(__FILE__) +  "/canvas_sync/jobs/*.rb"].each { |file| require file }
 Dir[File.dirname(__FILE__) +  "/canvas_sync/processors/*.rb"].each { |file| require file }
 Dir[File.dirname(__FILE__) +  "/canvas_sync/importers/*.rb"].each { |file| require file }
@@ -57,6 +59,9 @@ module CanvasSync
     graded_submissions
   ].freeze
 
+  JobBatches::ChainBuilder.register_chain_job(CanvasSync::Jobs::SyncTermsJob, :sub_jobs)
+  JobBatches::ChainBuilder.register_chain_job(CanvasSync::Jobs::BeginSyncChainJob, 0)
+
   class << self
     # Runs a standard provisioning sync job with no extra report types.
     # Terms will be synced first using the API. If you are syncing users/roles/admins
@@ -72,72 +77,9 @@ module CanvasSync
     #   and inserts it into the database. If an array of model names is provided then only those models will use legacy support.
     # @param account_id [Integer, nil] This optional parameter can be used if your Term creation and
     #   canvas_sync_client methods require an account ID.
-    def provisioning_sync(models, term_scope: nil, legacy_support: false, account_id: nil)
+    def provisioning_sync(models, **kwargs)
       validate_models!(models)
-      invoke_next(default_provisioning_report_chain(models, term_scope, legacy_support, account_id))
-    end
-
-    # Runs a report different from provisioning sync job with no extra report types.
-    #
-    # @param reports_mapping [Array<Hash>] An Array of hash that list the model and params with their report you
-    #   want to import:
-    #   [{model: 'submissions', report_name: 'my_report_name_csv', params: { "parameters[include_deleted]" => true } }, ...]
-    # @param term_scope [Symbol, nil] An optional symbol representing a scope that exists on the Term model.
-    #   The provisioning report will be run for each of the terms contained in that scope.
-    # @param account_id [Integer, nil] This optional parameter can be used if your Term creation and
-    #   canvas_sync_client methods require an account ID.
-    def simple_report_sync(reports_mapping, term_scope: nil, account_id: nil)
-      invoke_next(simple_report_chain(reports_mapping, term_scope, account_id))
-    end
-
-    # Runs a chain of ordered jobs
-    #
-    # See the README for usage and examples
-    #
-    # @param job_chain [Hash]
-    def process_jobs(job_chain)
-      invoke_next(job_chain)
-    end
-
-    def duplicate_chain(job_chain)
-      Marshal.load(Marshal.dump(job_chain))
-    end
-
-    # Invokes the next job in a chain of jobs.
-    #
-    # This should typically be called automatically by the gem where necessary.
-    #
-    # @param job_chain [Hash] A chain of jobs to execute
-    def invoke_next(job_chain, extra_options: {})
-      job_chain = job_chain.chain_data if job_chain.is_a?(JobChain)
-
-      return if job_chain[:jobs].empty?
-
-      # Make sure all job classes are serialized as strings
-      job_chain[:jobs].each { |job| job[:job] = job[:job].to_s }
-
-      duped_job_chain = Marshal.load(Marshal.dump(job_chain))
-      jobs = duped_job_chain[:jobs]
-      next_job = jobs.shift
-      next_job_class = next_job[:job].constantize
-      next_options = next_job[:options] || {}
-      next_options.merge!(extra_options)
-      next_job_class.perform_later(duped_job_chain, next_options)
-    end
-
-    def fork(job_log, job_chain, keys: [])
-      job_chain = job_chain.chain_data if job_chain.is_a?(JobChain)
-
-      duped_job_chain = Marshal.load(Marshal.dump(job_chain))
-      duped_job_chain[:global_options][:fork_path] ||= []
-      duped_job_chain[:global_options][:fork_keys] ||= []
-      duped_job_chain[:global_options][:fork_path] << job_log.job_id
-      duped_job_chain[:global_options][:fork_keys] << keys.map(&:to_s)
-      duped_job_chain[:global_options][:on_failure] ||= 'CanvasSync::Jobs::ForkGather.handle_branch_error'
-      sub_items = yield duped_job_chain
-      sub_count = sub_items.respond_to?(:count) ? sub_items.count : sub_items
-      job_log.fork_count = sub_count
-      sub_items
+      default_provisioning_report_chain(models, **kwargs).process!
     end
 
     # Given a Model or Relation, scope it down to items that should be synced
@@ -154,33 +96,6 @@ module CanvasSync
       scope
     end
 
-    # Syn any report to an specific set of models
-    #
-    # @param reports_mapping [Array<Hash>] List of reports with their specific model and params
-    # @param term_scope [String]
-    # @param account_id [Integer, nil] This optional parameter can be used if your Term creation and
-    #   canvas_sync_client methods require an account ID.
-    def simple_report_chain(reports_mapping, term_scope=nil, account_id=nil)
-      jobs = reports_mapping.map do |report|
-        {
-          job: CanvasSync::Jobs::SyncSimpleTableJob.to_s,
-          options: {
-            report_name: report[:report_name],
-            model: report[:model],
-            mapping: report[:model],
-            klass: report[:model].singularize.capitalize.to_s,
-            term_scope: term_scope,
-            params: report[:params]
-          }
-        }
-      end
-
-      global_options = {}
-      global_options[:account_id] = account_id if account_id.present?
-
-      JobChain.new(jobs: jobs, global_options: global_options)
-    end
-
     # Syncs terms, users/roles/admins if necessary, then the rest of the specified models.
     #
     # @param models [Array<String>]
@@ -191,7 +106,7 @@ module CanvasSync
     # @param account_id [Integer, nil] This optional parameter can be used if your Term creation and
     #   canvas_sync_client methods require an account ID.
     # @return [Hash]
-    def default_provisioning_report_chain(models, term_scope=nil, legacy_support=false, account_id=nil, options: {}) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity, Metrics/LineLength
+    def default_provisioning_report_chain(models, term_scope: nil, legacy_support: false, account_id: nil, updated_after: nil, options: {}) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity, Metrics/LineLength
       return unless models.present?
       models.map! &:to_s
       term_scope = term_scope.to_s if term_scope
@@ -210,64 +125,66 @@ module CanvasSync
         context_module_items: CanvasSync::Jobs::SyncContextModuleItemsJob,
       }.with_indifferent_access
 
-      jobs = []
+      root_chain = JobBatches::ChainBuilder.new(CanvasSync::Jobs::BeginSyncChainJob)
+      concurrent_root_chain = JobBatches::ChainBuilder.new(JobBatches::ConcurrentBatchJob)
+      root_chain << concurrent_root_chain
+      current_chain = concurrent_root_chain
+
       try_add_model_job = ->(model) {
         return unless models.include?(model)
-        jobs.push(job: model_job_map[model].to_s, options: options[model.to_sym] || {})
+        current_chain << { job: model_job_map[model].to_s, options: options[model.to_sym] || {} }
         models -= [model]
       }
 
       ##############################
-      # Pre provisioning report jobs
+      # General provisioning jobs (not term-scoped)
       ##############################
 
-      # Always sync Terms first
-      models.unshift('terms') unless models.include?('terms')
-      try_add_model_job.call('terms')
+      # Terms are always synced regardless of the models option
+      models -= ['terms']
 
-      # Accounts, users, roles, and admins are synced before provisioning because they cannot be scoped to term
+      # Accounts, users, roles, and admins cannot be scoped to term
       try_add_model_job.call('accounts')
 
       # These Models use the provisioning report, but are not term-scoped,
-      # so we sync them before to ensure work is not duplicated
+      # so we sync them outside of the term scoping to ensure work is not duplicated
       if term_scope.present?
         models -= (first_provisioning_models = models & ['users', 'pseudonyms'])
-        jobs.concat(
-          generate_provisioning_jobs(first_provisioning_models, options)
-        )
+        current_chain.insert(generate_provisioning_jobs(first_provisioning_models, options))
       end
 
       try_add_model_job.call('roles')
       try_add_model_job.call('admins')
-      pre_provisioning_jobs = jobs
 
       ###############################
-      # Post provisioning report jobs
+      # Per-term provisioning jobs
       ###############################
 
-      jobs = []
+      per_term_chain = JobBatches::ChainBuilder.new(model_job_map[:terms])
+      current_chain << per_term_chain
+      current_chain = per_term_chain
+
       try_add_model_job.call('assignments')
       try_add_model_job.call('submissions')
       try_add_model_job.call('assignment_groups')
       try_add_model_job.call('context_modules')
       try_add_model_job.call('context_module_items')
-      post_provisioning_jobs = jobs
+
+      current_chain.insert(
+        generate_provisioning_jobs(models, options, job_options: { term_scope: term_scope }, only_split: ['users'])
+      )
 
       ###############################
-      # Main provisioning job and queueing
+      # Wrap it all up
       ###############################
 
-      jobs = [
-        *pre_provisioning_jobs,
-        *generate_provisioning_jobs(models, options, job_options: { term_scope: term_scope }, only_split: ['users']),
-        *post_provisioning_jobs,
-      ]
-
-      global_options = { legacy_support: legacy_support }
+      global_options = { legacy_support: legacy_support, updated_after: updated_after }
       global_options[:account_id] = account_id if account_id.present?
       global_options.merge!(options[:global]) if options[:global].present?
 
-      JobChain.new(jobs: jobs, global_options: global_options)
+      root_chain.params[1] = global_options
+
+      root_chain
     end
 
     def group_by_job_options(model_list, options_hash, only_split: nil, default_key: :provisioning)