canvas_sync 0.16.4 → 0.21.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: ddf6a257acf95c92ba99a6d03c9bcdd99ba1a8ff
-  data.tar.gz: f10e8f91a225462545354a24daaf006a170bd90d
+SHA256:
+  metadata.gz: aecd0f3989ecb1b42c9f8c1c83274333eabe316bf05407cc2de7113e14590cb6
+  data.tar.gz: 104cbae017ac24e83b5ad357ed381c6af24577917ec1a494ef9456478874eea0
 SHA512:
-  metadata.gz: cde3a3a534316a0faa535347a6c71aa63ab46c5cc5d9c21b01512bb623bed7f6b34fc7da38de345fefcf238cc44f0423f849dd08aa8f57f292472838aa6f7d0c
-  data.tar.gz: 2e1b8217af038166f60afe4032f0f1ec806761fb81d5727aec57540919146335f7a28292e8e81ba87220eb2d687325b8145be798ed6a29b07aaad0cd6e3b56f4
+  metadata.gz: 8059b7b6fcadd5c0b8750b705881496dc63bb90a232e9d950618dff0682f5410ea9cc18a84b9a8a22ea4a051262fb2946c0c6c05a39e3a86871a6ee460f75f08
+  data.tar.gz: 2d1b85f6564c8916ae3b8a51290df8bf95d75156b2bb4b595466a502d78dd0f7f20260a14914ce482bd89af963b9b89e23207640c6df6a004616fcd9d0f62ff8

data/README.md

@@ -9,7 +9,7 @@ Add this line to your application's Gemfile:
 ```ruby
 gem 'canvas_sync'
 ```
-
+mount CanvasSync::Engine, at: '/canvas_sync'
 Models and migrations can be installed using the following generator:
 
 ```
@@ -34,6 +34,7 @@ The following custom reports are required for the specified models:
 - assignment_groups = "Assignment Group Export" (proserv_assignment_group_export_csv)
 - context_modules = "Professional Services Context Modules Report" (proserv_context_modules_csv)
 - context_module_items = "Professional Services Context Module Items Report" (proserv_context_module_items_csv)
+- content_migrations = "Professional Services Content Migrations Report" (proserv_content_migrations_csv)
 
 ## Prerequisites
 
@@ -49,18 +50,14 @@ Make sure you've setup sidekiq to work properly with ActiveJob as [outlined here
 
 If using apartment and sidekiq make sure you include the [apartment-sidekiq](https://github.com/influitive/apartment-sidekiq) gem so that the jobs are run in the correct tenant.
 
-## Basic Usage
-
-Your tool must have an `ActiveJob` compatible job queue adapter configured, such as DelayedJob or Sidekiq. Additionally, you must have a method called `canvas_sync_client` defined in an initializer that returns a Bearcat client for the Canvas instance you are syncing against. Example:
+### Live Events
+if enabling Live Events, the following additional dependencies are required:
+- If using Core/Data Services events: `httparty`, `json-jwt`
+- If using EventsManager/DejaVu events: `symmetric-encryption`
 
-```ruby
-# config/initializers/canvas_sync.rb
-def canvas_sync_client
-  Bearcat::Client.new(token: current_organization.settings[:api_token], prefix: current_organization.settings[:base_url])
-end
-```
+## Basic Usage
 
-(Having the client defined here means the sensitive API token doesn't have to be passed in plain text between jobs.)
+Your tool must have an `ActiveJob` compatible job queue adapter configured, such as DelayedJob or Sidekiq.
 
 Once that's done and you've used the generator to create your models and migrations you can run the standard provisioning sync:
 
@@ -82,6 +79,47 @@ If you pass in the optional `term_scope` the provisioning reports will be run fo
 
 Imports are inserted in bulk with [activerecord-import](https://github.com/zdennis/activerecord-import) so they should be very fast.
 
+### Live Events
+
+Ensure that
+```ruby
+mount CanvasSync::Engine, at: '/canvas_sync'
+```
+is added to your `routes.rb`. Configure `DataServices` or `EventsManager` to send events to `https://YOUR_APP/canvas_sync/api/v1/live_event` (if using `DataServices`, event must be signed).
+
+Uncomment `include CanvasSync::Concerns::LiveEventSync` and related lines in the appropriate models. (Some models provide some basic hooks to address a "typical" workflow).
+
+When Live Events are received, the corresponding model (if present) instance will receive `process_live_event(subtype, payload, metadata)` (where `subtype` is the event name w/o the model name - eg `user_created` => `created`). The default logic is to call `ApiSyncable` and update the model from the Canvas API. `process_live_event` can be overridden directly, or hooked with the usual Rails callbacks system (eg `before_process_live_event`).
+
+You can subscribe to Live Events outside of a model context using an intializer like so:
+```ruby
+CanvasSync::LiveEvents.subscribe(%w[Optional List of Events]) do |event|
+  # Your code here
+  # Note that this code is _not_ retried if it fails. If you need retries, use this block to trigger another Job.
+end
+```
+
+#### Event Provenance
+
+When using `EventsManager` events, events are verified as having come from a legitimate source by use of `SymmetricEncryption` (and thus `PRODUCTION_KEY1` will need to be set correctly when deployed).
+
+When using `DataServices`, CanvasSync uses the `DataServices` JWK to authenticate incoming events. CanvasSync is coded to default to the Prod & Beta JWK URL at https://8axpcl50e4.execute-api.us-east-1.amazonaws.com/main/jwks, but this can be overridden with the `DATASERVICES_JWK_URL` ENV variable.
+
+Additionally, when `PandaPal` is installed too, use `https://YOUR_APP/canvas_sync/api/v1/live_event?org=ORG_ID` instead. `CanvasSync` will automatically switch to the correct organization and will validate that the event was indeed from the correct Canvas instance. If you are not using `PandaPal`, you'll need to monkey-patch `CanvasSync::Api::V1::LiveEventsController#validate_tenant!`
+
+#### Legacy-Style Event Jobs
+
+CanvasSync also supports they legacy style of Event Handlers. In this design, properly-named classes are defined in the `::LiveEvents` module, such as (`class LiveEvents::UserCreatedEvent`). Any `ActiveJob` job is compatible, but CanvasSync also provides `CanvasSync::LiveEvents::BaseHandler` as a helpful base class.
+
+When present, these jobs will, per event type (eg `user_created`), override the default behavior, meaning that `subscribe` blocks and `process_live_event` and related callbacks will _not_ be called unless you call them. In other words: If you define `class LiveEvents::UserCreatedEvent` and also
+```ruby
+subscribe(%w[user_created user_updated]) do |event|
+  # ...
+end
+```
+the subscribe block (and User model) will receive `user_updated` events, but not `user_created` events.
+
+These jobs can also be generated from template using `bin/rails generate canvas_sync:install_live_events --events users,courses,etc`
 
 ## Advanced Usage
 
@@ -91,27 +129,58 @@ This gem also helps with syncing and processing other reports if needed. In orde
 - Integrate your reports with the `ReportStarter`
 - Tell the gem what jobs to run
 
+### `updated_after`
+An `updated_after` param may be passed when triggering a provision or making a chain:
+```ruby
+CanvasSync.default_provisioning_report_chain(
+  %i[list of models to sync], updated_after: false
+)
+```
+It may be one of the following values:
+* `false` - Will not apply any `updated_after` filtering to the requested reports
+* An ISO-8601 Date - Will pass the supplied date ad the `updated_after` param for the requested reports
+* `true` (Default) - Will use the start date of the last successful sync
+
+If `updated_after` is true, CanvasSync will, by default, perform a full sync every other Sunday.
+This logic can be customized by passing `full_sync_every` parameter.
+If you pass a date to `updated_after`, this logic will be disabled unless you explicitly pass a `full_sync_every` parameter.
+`full_sync_every` accepts the following format strings:
+- `15%` - Each sync will have a 15% chance of running a full sync
+- `10 days` - A full sync will be run every 10 days
+- `sunday` - A full sync will run every Sunday
+- `saturday/4` - A full sync will run every fourth Saturday
+
+#### Multiple Sync Chains
+If your app uses multiple Sync Chains, you may run into issues with the automatic `updated_after` and `full_sync_every` logic.
+You can fix this by using custom logic or by setting the `batch_genre` parameter when creating the Job Chain. Chains will only
+use chains of the same genre when computing `updated_after` and `full_sync_every`.
+
 ### Extensible chain
 It is sometimes desired to extend or customize the chain of jobs that are run with CanvasSync.
 This can be achieved with the following pattern:
 
 ```ruby
-job_chain = CanvasSync.default_provisioning_report_chain(
+chain = CanvasSync.default_provisioning_report_chain(
   %i[list of models to sync]
 )
 
-# CanvasSync forks the chain for each term within a term scope. The ForkGather Job can be used to unfork the chain.
-# For example multiple Terms are in the term_scope. CanvasSync syncs Accounts, Terms, and Users (if enabled) and then
-# forks the chain to sync other models (eg Course, Sections, etc.) per-Term.
-# ForkGather will wait until all the forked chains are complete before continuing.
-# TL;DR: Jobs placed after SyncProvisioningReportJob and before ForkGather will run once per Term per Sync;
-# Jobs placed before SyncProvisioningReportJob or after ForkGather will run once per Sync
-job_chain[:jobs] << { job: CanvasSync::Jobs::ForkGather, options: {} }
+# Add a custom job to the end of the chain.
+chain << { job: CanvasSyncCompleteWorker, args: [job.id], kwargs: { job_id: job.id } }
 
-# Add a custom job to the end of the chain. Custom jobs must accept 2 arguments (job_chain, options) and call CanvasSync.invoke_next(job_chain) when complete
-job_chain[:jobs] << { job: CanvasSyncCompleteWorker, options: { job_id: job.id } }
+chain.process!
 
-CanvasSync.invoke_next(job_chain)
+# The chain object provides a fairly extensive API:
+chain.insert({ job: SomeOtherJob, args: [], kwargs: {} }) # Adds the job to the end of the chain
+chain.insert_at(0, { job: SomeOtherJob }) # Adds the job to the beginning of the chain
+chain.insert({ job: SomeOtherJob }, after: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job right after the SyncTermsJob
+chain.insert({ job: SomeOtherJob }, before: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job right before the SyncTermsJob
+chain.insert({ job: SomeOtherJob }, with: 'CanvasSync::Jobs::SyncTermsJob') # Adds the job to be performed concurrently with the SyncTermsJob
+
+# Some Jobs (such as the SyncTermsJob) have a sub-chain for, eg, Courses.
+# chain.insert is aware of these sub-chains and will recurse into them when looking for a before:/after:/with: reference
+chain.insert({ job: SomeOtherJob }, after: 'CanvasSync::Jobs::SyncCoursesJob') # Adds the job to be performed after SyncCoursesJob (which is a sub-job of the terms job and is duplicated for each term in the term_scope:)
+# You can also retrieve the sub-chain like so:
+chain.get_sub_chain('CanvasSync::Jobs::SyncTermsJob')
 ```
 
 ### Processor
@@ -130,13 +199,12 @@ end
 
 You must implement a job that will enqueue a report starter for your report. (TODO: would be nice to make some sort of builder for this, so you just define the report and its params and then the gem runs it in a pre-defined job.)
 
-Let's say we have a custom Canvas report called "my_really_cool_report_csv". First, we would need to create a job class that will enqueue a report starter. To work with the `CanvasSync` interface, your class must accept 2 parameters: `job_chain`, and `options`.
+Let's say we have a custom Canvas report called "my_really_cool_report_csv". First, we would need to create a job class that will enqueue a report starter.
 
 ```ruby
 class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
-  def perform(job_chain, options)
+  def perform(options)
     super(
-      job_chain,
       'my_really_cool_report_csv', # Report name
       { "parameters[param1]" => true }, # Report parameters
       MyCoolProcessor.to_s, # Your processor class as a string
@@ -148,69 +216,41 @@ end
 
 You can also see examples in `lib/canvas_sync/jobs/sync_users_job.rb` and `lib/canvas_sync/jobs/sync_provisioning_report.rb`.
 
-### Start the jobs
+### Batching
 
-The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to run. The job chain must be formatted like:
+The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable.
 
-```ruby
-{
-  jobs: [
-    { job: JobClass, options: {} },
-    { job: JobClass2, options: {} }
-  ],
-  global_options: {}
-}
-```
+### Mapping Overrides
 
-Here is an example that runs our new report job first followed by the builtin provisioning job:
+Overrides are useful for two scenarios:
 
-```ruby
-job_chain = {
-  jobs: [
-    { job: MyReallyCoolReportJob, options: {} },
-    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
-  ],
-  global_options: {}
-}
+- You have an existing application where the column names do not match up with what CanvasSync expects
+- You want to sync some other column in the report that CanvasSync is not configured to sync
 
-CanvasSync.process_jobs(job_chain)
-```
+Mappings can be modified by editing the Model class like such:
+```ruby
+class User < ApplicationRecord
+  include CanvasSync::Record
 
-What if you've got some other job that you want run that doesn't deal with a report? No problem! Just make sure you call `CanvasSync.invoke_next` at the end of your job. Example:
+  sync_mapping(reset: false) do # `reset: false` is the default
+    # The mapping can be totally cleared with `reset: true` in the `sync_mapping` call, or like such:
+    reset_links
 
+    # Add a new column:
+    link_column :column_in_report => :column_in_database, type: :datetime
 
-```ruby
-class SomeRandomJob < CanvasSync::Job
-  def perform(job_chain, options)
-    i_dunno_do_something!
+    # If the column name on the report and in the DB are the same, a shorthand can be used:
+    link_column :omit_from_final_grade, type: :datetime
 
-    CanvasSync.invoke_next(job_chain)
+    # If the defaults define a column you don't want synced, you can remove it from the mapping:
+    unlink_column :column_in_database
   end
-end
 
-job_chain = {
-  jobs: [
-    { job: SomeRandomJob, options: {} },
-    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
-  ],
-  global_options: {}
-}
-
-CanvasSync.process_jobs(job_chain)
+  # ...
+end
 ```
 
-### Batching
-
-The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable.
-
-### Mapping Overrides
-
-Overrides are useful for two scenarios:
-
-- You have an existing application where the column names do not match up with what CanvasSync expects
-- You want to sync some other column in the report that CanvasSync is not configured to sync
-
-In order to create an override, place a file called `canvas_sync_provisioning_mapping.yml` in your Rails `config` directory. Define the tables and columns you want to override using the following format:
+You can also create a file called `canvas_sync_provisioning_mapping.yml` in your Rails `config` directory. However, this approach requires you to re-specify the complete table in order to modify a table. Define the tables and columns you want to override using the following format:
 
 ```ruby
 users:
@@ -257,6 +297,87 @@ class CanvasSyncModel < ApplicationRecord
 end
 ```
 
+### Job Batching
+CanvasSync adds a `CanvasSync::JobBatches` module. It adds Sidekiq/sidekiq-batch like support for Job Batches.
+It integrates automatically with both Sidekiq and ActiveJob. The API is highly similar to the Sidekiq-batch implementation,
+documentation for which can be found at https://github.com/mperham/sidekiq/wiki/Batches
+
+A batch can be created using `Sidekiq::Batch` or `CanvasSync::JobBatching::Batch`.
+
+Also see `canvas_sync/jobs/begin_sync_chain_job`, `canvas_sync/Job_batches/jobs/serial_batch_job`, or `canvas_sync/Job_batches/jobs/concurrent_batch_job` for example usage.
+
+Example:
+```ruby
+batch = CanvasSync::JobBatches::Batch.new
+batch.description = "Some Batch" # Optional, but can be useful for debugging
+
+batch.on(:complete, "SomeClass.on_complete", kw_param: 1)
+batch.on(:success, "SomeClass.on_success", some_param: 'foo')
+
+# Add context to the batch. Can be accessed as batch_context on any jobs within the batch.
+# Nested Batches will have their contexts merged
+batch.context = {
+  some_value: 'blah',
+}
+
+batch.jobs do
+  # Enqueue jobs like normal
+end
+```
+
+#### Job Pools
+A job pool is like a custom Sidekiq Queue. You can add jobs to it and it will empty itself out into one of the actual queues.
+However, it adds some options for tweaking the logic:
+- `concurrency` (default: `nil`) - Define how many jobs from the pool can run at once.
+- `order` (default: `fifo`) - Define how the pool will empty itself
+  - `fifo` - First-In First-Out, a traditional queue
+  - `lifo` - Last-In First-Out
+  - `random` - Pluck and run jobs in random order
+  - `priority` - Execute jobs in a priority order (NB: Due to Redis, this priority-random, meaning that items with the same priority will be run in random order, not fifo)
+- `clean_when_empty` (default: `true`) - Automatically clean the pool when it is empty
+- `on_failed_job` (default `:wait`) - If a Job fails, should the pool `:continue` and still enqueue the next job or `:wait` for the job to succeed
+
+Example:
+```ruby
+pool = CanvasSync::JobBatches::Pool.new(concurrency: 4, order: :priority, clean_when_empty: false)
+pool_id = pool.pid
+
+# Add a job to the pool
+pool << {
+  job: SomeJob, # The Class of a ActiveJob Job or Sidekiq Worker
+  args: [1, 2, 3], # Array of params to pass th e Job
+  kwargs: {},
+  priority: 100, # Only effective if order=:priority, higher is higher
+}
+
+# Add many jobs to the pool
+pool.add_jobs([
+  {
+    job: SomeJob, # The Class of a ActiveJob Job or Sidekiq Worker
+    args: [1, 2, 3], # Array of params to pass th e Job
+    kwargs: {},
+    priority: 100, # Only effective if order=:priority, higher is higher
+  },
+  # ...
+])
+
+# ...Later
+CanvasSync::JobBatches::Pool.from_pid(pool_id).cleanup_redis
+```
+
+### Custom Bearcat Instance
+You can define a global `canvas_sync_client` method to return a Bearcat Client instance for CanvasSync to use:
+```ruby
+# config/initializers/canvas_sync.rb
+def canvas_sync_client
+  Bearcat::Client.new(token: current_organization.settings[:api_token], prefix: current_organization.settings[:base_url])
+end
+```
+
+(Having the client defined here means the sensitive API token doesn't have to be passed in plain text between jobs.)
+
+This used to be required, but when both CanvasSync and PandaPal are up to date, this is defined automagically.
+
 
 ## Legacy Support
 
@@ -297,7 +418,7 @@ If you want your own jobs to also log to the table all you have to do is have yo
 
 If you want to be able to utilize the `CanvasSync::JobLog` without `ActiveJob` (so you can get access to `Sidekiq` features that `ActiveJob` doesn't support), then add the following to an initializer in your Rails app:
 
-```
+```ruby
 Sidekiq.configure_server do |config|
   config.server_middleware do |chain|
     chain.add CanvasSync::Sidekiq::Middleware
@@ -308,87 +429,26 @@ end
 ## Syncronize different reports
 CanvasSync provides the functionality to import data from other reports into an specific table.
 
-This can be achived by using the followin method
-
-```ruby
-CanvasSync.provisioning_sync(<array of models to sync>, term_scope: <optional term scope>)
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: <report name>,
-        model: <model to sync>,
-        params: <hash with the require parameters the report needs to sync>
-      },
-      {
-        report_name: <report name>,
-        model: <model to sync>,
-        params: <hash with the require parameters the report needs to sync>
-      },
-      ...
-    ],
-    term_scope: <optional term scope>
-  )
-```
-
-Example:
-
-```ruby
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'users',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[users]" => true
-        }
-      },
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'accounts',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[accounts]" => true
-        }
-      }
-    ]
-  )
-```
-
-Example with the term_scope active:
+This can be achieved by using the following method
 
 ```ruby
-CanvasSync
-  .simple_report_sync(
-    [
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'sections',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[sections]" => true
-        }
-      },
-      {
-        report_name: 'proservices_provisioning_csv',
-        model: 'courses',
-        params: {
-          "parameters[include_deleted]" => true,
-          "parameters[courses]" => true
-        }
-      }
-    ],
-    term_scope: 'active'
-  )
+chain = CanvasSync.default_provisioning_report_chain
+chain << {
+  job: CanvasSync::Jobs::SyncSimpleTableJob,
+  options: {
+    report_name: <report name>,
+    model: <model to sync>,
+    params: <hash with the require parameters the report needs to sync>
+  },
+}
+chain.process!
 ```
 
 ## Configuration
 
 You can configure CanvasSync settings by doing the following:
 
-```
+```ruby
 CanvasSync.configure do |config|
   config.classes_to_only_log_errors_on << "ClassToOnlyLogErrorsOn"
 end
@@ -398,6 +458,29 @@ Available config options (if you add more, please update this!):
 
 * `config.classes_to_only_log_errors_on` - use this if you are utilizing the `CanvasSync::JobLog` table, but want certain classes to only persist in the `job_logs` table if an error is encountered. This is useful if you've got a very frequently used job that's filling up your database, and only really care about tracking failures.
 
+## Global Options
+You can pass in global_options to a job chain.  Global options are added to the batch_context and referenced by
+various internal processes.
+
+Pass global options into a job chain, using the options param nested in a :global key.
+`options: { global: {...} }`
+
+report_timeout (integer):             Number of days until a Canvas report should timeout.  Default is 1.
+report_compilation_timeout (integer): Number of days until a Canvas report should timeout.  Default is 1 hour.
+                                      You can likely pass a float to achieve sub-day timeouts, but not tested.
+report_max_tries (integer):           The number of times to attempt a report before giving up.  A report is considered failed
+                                      if it has an 'error' status in Canvas or is deleted.
+
+This is an example job chain with global options:
+```ruby
+job_chain = CanvasSync.default_provisioning_report_chain(
+  MODELS_TO_SYNC,
+  term_scope: :active,
+  full_sync_every: 'sunday',
+  options: { global: { report_timeout: 2 } }
+)
+```
+
 ## Handling Job errors
 
 If you need custom handling for when a CanvasSync Job fails, you can add an `:on_failure` option to you Job Chain's `:global_options`.
@@ -421,7 +504,6 @@ class CanvasSyncStarterWorker
         }
       }
     )
-    CanvasSync.invoke_next(job_chain)
   end
 
   def self.handle_canvas_sync_error(error, **options)
@@ -436,7 +518,9 @@ Re-running the generator when there's been a gem change will give you several ch
 
 Additionally, if there have been schema changes to an existing model you may have to run your own migration to bring it up to speed.
 
-If you make updates to the gem please add any upgrade instructions here.
+Also see `CHANGELOG.md`.
+
+If you make updates to the gem please add any upgrade instructions to `CHANGELOG.md`.
 
 ## Integrating with existing applications
 

data/app/controllers/{api → canvas_sync/api}/v1/health_check_controller.rb

@@ -1,4 +1,4 @@
-module Api
+module CanvasSync::Api
   module V1
     class HealthCheckController < ActionController::Base
       def ping

data/app/controllers/canvas_sync/api/v1/live_events_controller.rb

@@ -0,0 +1,122 @@
+module CanvasSync::Api
+  module V1
+    class LiveEventsController < ActionController::Base
+      around_action :switch_tenant
+
+      def process_event
+        if params[:payload].present?
+          process_eventsmanager_event
+        else
+          process_dataservices_event
+        end
+      rescue => e
+        Rails.logger.error("Live Events Error: #{e.message} - #{e.backtrace}")
+        render json: { error: "Live Events Error: #{e.message}" }, status: 422
+      end
+
+      private
+
+      def process_eventsmanager_event
+        event = SymmetricEncryption.decrypt(params[:payload])
+        event = JSON.parse(event).with_indifferent_access
+
+        event[:metadata] = event[:attributes]
+        event.delete(:attributes)
+
+        Rails.logger.debug("Processing event type: #{event['metadata']['event_name']}")
+        Rails.logger.debug("Payload: #{event}")
+
+        validate_tenant!(event.dig(:metadata, :root_account_uuid))
+        transform_ids!(event)
+        event[:via] = "eventsmanager"
+
+        dispatch_event(event)
+
+        head :ok
+      end
+
+      def process_dataservices_event
+        require "json/jwt"
+        require "httparty"
+
+        event = nil
+
+        # Only allow unsigned events during devleopment
+        if Rails.env.development?
+          event = JSON.parse(request.raw_post) rescue nil
+        end
+
+        event ||= JSON::JWT.decode(params[:_json], dataservices_jwks)
+        event = event.to_h.with_indifferent_access
+
+        Rails.logger.debug("Processing event type: #{event['metadata']['event_name']}")
+        Rails.logger.debug("Payload: #{event}")
+
+        validate_tenant!(event.dig(:metadata, :root_account_uuid))
+        transform_ids!(event)
+        event[:via] = "dataservices"
+
+        dispatch_event(event)
+
+        head :ok
+      end
+
+      def dispatch_event(event)
+        CanvasSync::LiveEvents::ProcessEventJob.perform_later(event)
+      end
+
+      # Live events will use a canvas global ID (cross shard) for any ID's provided. This method will return the local ID.
+      def local_canvas_id(id)
+        # TODO: Don't apply this to cross-shard entries
+        id.to_i % 10_000_000_000_000
+      end
+
+      def transform_ids!(data)
+        transformed = {}
+        data.each do |k, v|
+          if k.ends_with?("_id") && (v.is_a?(String) || v.is_a?(Numeric))
+            transformed["sharded_#{k}"] = v
+            transformed[k] = local_canvas_id(v) rescue v
+          elsif v.is_a?(Hash)
+            transform_ids!(v)
+          end
+        end
+        data.merge!(transformed)
+      end
+
+      def dataservices_jwks
+        require "httparty"
+
+        jwk_json = Rails.cache.fetch("canvas_sync/dataservices_jwks", expires_in: 24.hours) do
+          jkws_url = ENV["DATASERVICES_JWK_URL"].presence || "https://8axpcl50e4.execute-api.us-east-1.amazonaws.com/main/jwks"
+          response = HTTParty.get(jkws_url)
+          JSON.parse(response.body)
+        end
+
+        JSON::JWK::Set.new(jwk_json)
+      end
+
+      def validate_tenant!(event_uuid)
+        if defined?(PandaPal)
+          root_info = current_organization.root_account_info
+          root_uuid = root_info[:uuid]
+
+          if !root_uuid.present? || root_uuid != event_uuid
+            render json: { error: "Invalid Organization/UUID" }, status: 403
+            return
+          end
+        else
+          raise "No way to validate LiveEvent is genuinely from the correct Canvas instance! Monkey-patch CanvasSync::Api::V1::LiveEventsController#validate_tenant!"
+        end
+      end
+
+      def switch_tenant(&block)
+        if defined?(PandaPal) && (org = params[:organization] || params[:org]).present?
+          Apartment::Tenant.switch(PandaPal::Organization.find(org).name, &block)
+        else
+          yield
+        end
+      end
+    end
+  end
+end

data/app/models/canvas_sync/sync_batch.rb

@@ -0,0 +1,5 @@
+module CanvasSync
+  class SyncBatch < ApplicationRecord
+    serialize :job_arguments, Array
+  end
+end