canvas_sync 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d6178b9db26d809e2e35b5c4d38187d2ceb14018
+  data.tar.gz: ec84b34894e5b40e27666cdb0946053a355cf422
+SHA512:
+  metadata.gz: 4ee7ea65f6c785d05559f52b841483ea2a316509373feedc7c29a35599bfe99d20214ca3a2b2116d1ef503d366a03ec6382d59eb0636e361138cb6378ef6afe1
+  data.tar.gz: aaf71eaaf57f0ff80e7c5e92a22f3a3340e57358ab9a1042f1e821ad15dacbd0b401fd93199bbd0c0e9bbb3ff5540c223ce512be15315b5ad71dd43591e92875

data/README.md

@@ -0,0 +1,178 @@
+# CanvasSync
+
+This gem is intended to facilitate fast and easy syncing of Canvas data.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'canvas_sync'
+```
+
+Models and migrations can be installed using the following generator:
+
+```
+bin/rails generate canvas_sync:install --models users,terms,courses
+```
+
+Use the `--models` option to specify what models you would like installed. This will add both the model files and their corresponding migrations. If you'd like to install all the models that `CanvasSync` supports then specify `--models all`.
+
+For a list of currently supported models, see `CanvasSync::SUPPORTED_MODELS`.
+
+Additionally, your Canvas instance must have the "Proserv Provisioning Report" enabled.
+
+## Docs
+
+Docs can be generated using [yard](https://yardoc.org/). To view the docs:
+
+- Clone this gem's repository
+- `bundle install`
+- `yard server --reload`
+
+The yard server will give you a URL you can visit to view the docs.
+
+## Basic Usage
+
+Your tool must have an `ActiveJob` compatible job queue adapter configured, such as DelayedJob or Sidekiq. Once that's done and you've used the generator to create your models and migrations you can run the standard provisioning sync:
+
+```ruby
+CanvasSync.provisioning_sync(<canvas base url>, <canvas api token>, <array of models to sync>, <optional term scope>)
+```
+
+Example:
+
+```ruby
+CanvasSync.provisioning_sync("https://pseng.instructure.com", "12345", ['users', 'courses'], :active)
+```
+
+This will kick off a string of jobs to sync your specified models.
+
+If you pass in the optional `term_scope` the provisioning reports will be run for only the terms returned by that scope. The scope must be defined on your `Term` model. (A sample one is provided in the generated `Term`.)
+
+Imports are inserted in bulk with [https://github.com/zdennis/activerecord-import](activerecord-import) so they should be very fast.
+
+## Advanced Usage
+
+This gem also helps with syncing and processing other reports if needed. In order to do so, you must:
+
+- Define a `Processor` class that implements a `process` method for handling the results of the report
+- Integrate your reports with the `ReportStarter`
+- Tell the gem what jobs to run
+
+### Processor
+
+Your processor class must implement a `process` class method that receives a `report_file_path` and a hash of `options`. (See the `CanvasSync::Processors::ProvisioningReportProcessor` for an example.) The gem handles the work of enqueueing and downloading the report and then passes the file path to your class to process as needed. A simple example might be:
+
+```ruby
+class MyCoolProcessor
+  def self.process(report_file_path, options)
+    puts "I downloaded a report to #{report_file_path}! Isn't that neat!"
+  end
+end
+```
+
+### Report starter
+
+You must implement a job that will enqueue a report starter for your report. (TODO: would be nice to make some sort of builder for this, so you just define the report and its params and then the gem runs it in a pre-defined job.)
+
+Let's say we have a custom Canvas report called "my_really_cool_report_csv". First, we would need to create a job class that will enqueue a report starter. To work with the `CanvasSync` interface, your class must accept 4 parameters: `canvas_base_url`, `canvas_api_token`, `job_chain`, and `options`.
+
+```ruby
+class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
+  def perform(canvas_base_url, canvas_api_token, job_chain, options)
+    super(
+      canvas_base_url,
+      canvas_api_token,
+      job_chain,
+      'my_really_cool_report_csv', # Report name
+      { "parameters[param1]" => true }, # Report parameters
+      MyCoolProcessor, # Your processor class
+      options
+    )
+  end
+end
+```
+
+You can also see examples in `lib/canvas_sync/jobs/sync_users_job.rb` and `lib/canvas_sync/jobs/sync_provisioning_report.rb`.
+
+### Start the jobs
+
+The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to run. The job chain must be formatted like:
+
+```ruby
+{
+  jobs: [
+    { job: JobClass, options: {} },
+    { job: JobClass2, options: {} }
+  ],
+  options: {}
+}
+```
+
+Here is an example that runs our new report job first followed by the builtin provisioning job:
+
+```
+job_chain = {
+  jobs: [
+    { job: MyReallyCoolReportJob, options: {} },
+    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
+  ],
+  options: {}
+}
+
+CanvasSync.process_jobs("https://pseng.instructure.com", "my-api-token", job_chain)
+```
+
+What if you've got some other job that you want run that doesn't deal with a report? No problem! Just make sure you call `CanvasSync.invoke_next` at the end of your job. Example:
+
+
+```
+class SomeRandomJob < CanvasSync::Jobs::ApplicationJob
+  def perform(canvas_base_url, canvas_api_token, job_chain, options)
+    i_dunno_do_something!
+
+    CanvasSync.invoke_next(canvas_base_url, canvas_api_token, job_chain)
+  end
+end
+
+job_chain = {
+  jobs: [
+    { job: SomeRandomJob, options: {} },
+    { job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
+  ],
+  options: {}
+}
+
+CanvasSync.process_jobs("https://pseng.instructure.com", "my-api-token", job_chain)
+```
+
+### Batching
+
+The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable if needed.
+
+## Upgrading
+
+Re-running the generator when there's been a gem change will give you several choices if it detects conflicts between your local files and the updated generators. You can either view a diff or allow the generator to overwrite your local file. In most cases you may just want to add the code from the diff yourself so as not to break any of your customizations.
+
+TODO - if you make any breaking changes make sure you include upgrade instructions here.
+
+## Integrating with existing applications
+
+In order for this to work properly your database tables will need to have at least the columns defined in this gem. (Adding additional columns is fine.) As such, you may need to run some migrations to rename existing columns or add missing ones. The generator only works well in a situation where that table does not already exist. Take a look at the migration templates in `lib/canvas_sync/generators/templates` to see what you need.
+
+## Development
+
+When adding to or updating this gem, make sure you do the following:
+
+- If you modify a table that's already in the released version of the gem please write *new migrations* rather than modifying the existing ones. This will allow people to more easily upgrade.
+- Update the yardoc comments where necessary, and confirm the changes by running `yardoc --server`
+- Write specs
+- If you modify the model or migration templates, run `bundle exec rake update_test_schema` to update them in the Rails Dummy application (and commit those changes)
+
+## TODO
+
+Currently in a pre-alpha like state.
+
+- Add better support for parallel job execution with job dependencies
+- Add some sort of `JobLog` model for tracking parallel execution and job metadata

data/Rakefile

@@ -0,0 +1,23 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "open3"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec
+
+desc 'This updates the models and migrations used by the testing Rails Dummy app and should be run whenever those are updated.'
+task :update_test_schema do
+  puts "Updating test models and migration files..."
+  stream_command("spec/dummy/bin/rails generate canvas_sync:install --models all --force")
+  puts "Updating the test database and schema..."
+  stream_command("cd spec/dummy; bundle exec rake db:drop; bundle exec rake db:migrate")
+end
+
+def stream_command(cmd)
+  Open3.popen2e(cmd) do |stdin, stdout_stderr, wait_thr|
+    while line = stdout_stderr.gets
+      puts line
+    end
+  end
+end

data/lib/canvas_sync.rb

@@ -0,0 +1,96 @@
+require "bearcat"
+require "canvas_sync/version"
+
+require "canvas_sync/jobs/application_job"
+require "canvas_sync/jobs/report_starter"
+require "canvas_sync/jobs/report_checker"
+require "canvas_sync/jobs/report_processor_job"
+require "canvas_sync/jobs/sync_provisioning_report_job"
+require "canvas_sync/jobs/sync_terms_job"
+require "canvas_sync/jobs/sync_users_job"
+
+Dir[File.dirname(__FILE__) +  "/canvas_sync/processors/*.rb"].each {|file| require file }
+Dir[File.dirname(__FILE__) +  "/canvas_sync/importers/*.rb"].each {|file| require file }
+Dir[File.dirname(__FILE__) +  "/canvas_sync/generators/*.rb"].each {|file| require file }
+
+module CanvasSync
+  SUPPORTED_MODELS = %w(users courses terms enrollments sections)
+
+  # Runs a standard provisioning sync job with no extra report types.
+  # Terms will be synced first using the API. If you are syncing users
+  # and have also specified a Term scope, Users will by synced first, before
+  # every other model (as Users are never scoped to Term).
+  #
+  # @param canvas_base_url [String] e.g., "https://pseng.instructure.com"
+  # @param canvas_api_token [String] A Canvas API token for the specified Canvas instance
+  # @param models [Array<String>] A list of models to sync. e.g., ['users', 'courses'].
+  #   must be one of SUPPORTED_MODELS
+  # @param term_scope [Symbol, nil] An optional symbol representing a scope that exists on the Term model.
+  #   The provisioning report will be run for each of the terms contained in that scope.
+  def self.provisioning_sync(canvas_base_url, canvas_api_token, models, term_scope=nil)
+    validate_models!(models)
+    invoke_next(canvas_base_url, canvas_api_token, default_provisioning_report_chain(models, term_scope))
+  end
+
+  # Runs a chain of ordered jobs
+  #
+  # See the README for usage and examples
+  #
+  # @param canvas_base_url [String]
+  # @param canvas_api_token [String]
+  # @param job_chain [Hash]
+  def self.process_jobs(canvas_base_url, canvas_api_token, job_chain)
+    invoke_next(canvas_base_url, canvas_api_token, job_chain)
+  end
+
+  # Invokes the next job in a chain of jobs.
+  #
+  # This should typically be called automatically by the gem where necessary.
+  #
+  # @param canvas_base_url [String] e.g., "https://pseng.instructure.com"
+  # @param canvas_api_token [String] A Canvas API token for the specified Canvas instance
+  # @param job_chain [Hash] A chain of jobs to execute
+  def self.invoke_next(canvas_base_url, canvas_api_token, job_chain)
+    return if job_chain[:jobs].empty?
+
+    # Make sure all job classes are serialized as strings
+    job_chain[:jobs].each { |job| job[:job] = job[:job].to_s }
+
+    duped_job_chain = Marshal.load(Marshal.dump(job_chain))
+    jobs = duped_job_chain[:jobs]
+    next_job = jobs.shift
+    next_job_class = next_job[:job].constantize
+    next_job_class.perform_later(canvas_base_url, canvas_api_token, duped_job_chain, next_job[:options])
+  end
+
+  # Syncs terms, users if necessary, then the rest of the specified models.
+  #
+  # @param models [Array<String>]
+  # @param term_scope [String]
+  # @return [Hash]
+  def self.default_provisioning_report_chain(models, term_scope=nil)
+    term_scope = term_scope.to_s if term_scope
+
+    # Always sync Terms first
+    jobs = [{ job: CanvasSync::Jobs::SyncTermsJob.to_s, options: {} }]
+
+    if models.include?('users') && term_scope.present?
+      # Sync all users first when scoping by term, because users cannot be scoped to term
+      jobs.push({ job: CanvasSync::Jobs::SyncUsersJob.to_s, options: {} })
+      models = models - ['users']
+    end
+
+    jobs.push({ job: CanvasSync::Jobs::SyncProvisioningReportJob.to_s, options: { term_scope: term_scope, models: models } })
+
+    { jobs: jobs, options: {} }
+  end
+
+  private
+
+  def self.validate_models!(models)
+    invalid = models - SUPPORTED_MODELS
+    if invalid.length > 0
+      raise "Invalid model(s) specified: #{invalid.join(', ')}. Only #{SUPPORTED_MODELS.join(', ')} are supported."
+    end
+  end
+end

data/lib/canvas_sync/generators/install_generator.rb

@@ -0,0 +1,54 @@
+require "rails/generators"
+require "rails/generators/migration"
+
+module CanvasSync
+  class InstallGenerator < Rails::Generators::Base
+    include Rails::Generators::Migration
+    source_root File.expand_path('../templates', __FILE__)
+    class_option :models, type: :string, required: true
+
+    def self.next_migration_number(path)
+      next_migration_number = current_migration_number(path) + 1
+      ActiveRecord::Migration.next_migration_number(next_migration_number)
+    end
+
+    def autogenerated_model_warning
+      <<-HERE.strip_heredoc
+        #
+        # AUTO GENERATED MODEL
+        # This model was auto generated by the CanvasSync Gem.
+        # You can customize it as needed, but make sure you test
+        # any changes you make to the auto generated methods.
+        #
+      HERE
+    end
+
+    def autogenerated_migration_warning
+      <<-HERE.strip_heredoc
+        #
+        # AUTO GENERATED MIGRATION
+        # This migration was auto generated by the CanvasSync Gem.
+        # You can add new columns to this table, but removing or
+        # re-naming ones created here may break Canvas Syncing.
+        #
+      HERE
+    end
+
+    # Generates the specified models and migrations. Invoke with:
+    #
+    # bin/rails generate canvas_sync:install --models users,courses
+    #
+    # Install all models and migrations with:
+    #
+    # bin/rails generate canvas_sync:install --models all
+    def generate_migrations_and_models
+      models = options['models'] == 'all' ? CanvasSync::SUPPORTED_MODELS : options['models'].split(',')
+      CanvasSync.validate_models!(models)
+
+      models.each do |model|
+        migration_template "create_#{model}.rb", "db/migrate/create_#{model}.rb"
+        template "#{model.singularize}.rb", "app/models/#{model.singularize}.rb"
+      end
+    end
+  end
+end

data/lib/canvas_sync/generators/templates/course.rb

@@ -0,0 +1,8 @@
+<%= autogenerated_model_warning %>
+
+class Course < ApplicationRecord
+  validates :canvas_course_id, uniqueness: true, presence: true
+  belongs_to :term, foreign_key: :canvas_term_id, primary_key: :canvas_term_id, optional: true
+  has_many :enrollments, primary_key: :canvas_course_id, foreign_key: :canvas_course_id
+  has_many :sections, primary_key: :canvas_course_id, foreign_key: :canvas_course_id
+end

data/lib/canvas_sync/generators/templates/create_courses.rb

@@ -0,0 +1,21 @@
+<%= autogenerated_migration_warning %>
+
+class CreateCourses < ActiveRecord::Migration[5.1]
+  def change
+    create_table :courses do |t|
+      t.bigint :canvas_course_id, null: false
+      t.string :sis_id
+      t.string :short_name
+      t.string :long_name
+      t.integer :canvas_account_id
+      t.integer :canvas_term_id
+      t.integer :term_sis_id
+      t.datetime :start_date
+      t.datetime :end_date
+
+      t.timestamps
+    end
+
+    add_index :courses, :canvas_course_id, unique: true
+  end
+end

data/lib/canvas_sync/generators/templates/create_enrollments.rb

@@ -0,0 +1,26 @@
+<%= autogenerated_migration_warning %>
+
+class CreateEnrollments < ActiveRecord::Migration[5.1]
+  def change
+    create_table :enrollments do |t|
+      t.bigint :canvas_enrollment_id, null: false
+      t.bigint :canvas_course_id
+      t.string :course_sis_id
+      t.bigint :canvas_user_id
+      t.string :user_sis_id
+      t.string :role
+      t.integer :role_id
+      t.bigint :canvas_section_id
+      t.string :section_sis_id
+      t.string :status
+      t.string :base_role_type
+
+
+      t.timestamps
+    end
+
+    add_index :enrollments, :canvas_enrollment_id, unique: true
+    add_index :enrollments, :canvas_course_id
+    add_index :enrollments, :canvas_user_id
+  end
+end

data/lib/canvas_sync/generators/templates/create_sections.rb

@@ -0,0 +1,20 @@
+<%= autogenerated_migration_warning %>
+
+class CreateSections < ActiveRecord::Migration[5.1]
+  def change
+    create_table :sections do |t|
+      t.bigint :canvas_section_id, null: false
+      t.string :sis_id
+      t.bigint :canvas_course_id
+      t.string :name
+      t.string :status
+      t.datetime :start_date
+      t.datetime :end_date
+
+      t.timestamps
+    end
+
+    add_index :sections, :canvas_section_id, unique: true
+    add_index :sections, :canvas_course_id
+  end
+end

data/lib/canvas_sync/generators/templates/create_terms.rb

@@ -0,0 +1,18 @@
+<%= autogenerated_migration_warning %>
+
+class CreateTerms < ActiveRecord::Migration[5.1]
+  def change
+    create_table :terms do |t|
+      t.integer :canvas_term_id, null: false
+      t.string :name
+      t.datetime :start_at
+      t.datetime :end_at
+      t.string :workflow_state
+      t.integer :grading_period_group_id
+      t.string :sis_id
+
+      t.timestamps
+    end
+    add_index :terms, :canvas_term_id, unique: true
+  end
+end