canvas_sync 0.3.1 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +62 -51
- data/lib/canvas_sync/importers/bulk_importer.rb +2 -2
- data/lib/canvas_sync/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9ef8df28c558a21b5843f195f807b091efcc2cd0
|
4
|
+
data.tar.gz: '0549a9d1691f509116fbc136552b6ed2847ee8e1'
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d4712ea7567083b38968a4256e419546e126466886f1b81898bb018943f01c4471c326f5364d65e390b79ccc8fd26f8240868784847fce9f7748f082e4ed8fe2
|
7
|
+
data.tar.gz: ccd1e2beecb5651f511616ac56848e9282b108188831ca7ee811723ae7d14fadc736fe1a5bc605b7bde9aa3ad82804aaf711a7716718e29a6bc3cae322b0d8f5
|
data/README.md
CHANGED
@@ -28,15 +28,19 @@ For a list of currently supported models, see `CanvasSync::SUPPORTED_MODELS`.
|
|
28
28
|
|
29
29
|
Additionally, your Canvas instance must have the "Proserv Provisioning Report" enabled.
|
30
30
|
|
31
|
-
##
|
31
|
+
## Prerequisites
|
32
32
|
|
33
|
-
|
33
|
+
### Postgres
|
34
34
|
|
35
|
-
|
36
|
-
- `bundle install`
|
37
|
-
- `yard server --reload`
|
35
|
+
The bulk inserting is made possible by using a Postgres upsert. Beause of this, you need to be using **Postgres 9.5** or above.
|
38
36
|
|
39
|
-
|
37
|
+
### Sidekiq
|
38
|
+
|
39
|
+
Make sure you've setup sidekiq to work properly with ActiveJob as [outlined here](https://github.com/mperham/sidekiq/wiki/Active-Job).
|
40
|
+
|
41
|
+
### Apartment
|
42
|
+
|
43
|
+
If using apartment and sidekiq make sure you include the [apartment-sidekiq](https://github.com/influitive/apartment-sidekiq) gem so that the jobs are run in the correct tenant.
|
40
44
|
|
41
45
|
## Basic Usage
|
42
46
|
|
@@ -57,7 +61,7 @@ Once that's done and you've used the generator to create your models and migrati
|
|
57
61
|
CanvasSync.provisioning_sync(<array of models to sync>, term_scope: <optional term scope>)
|
58
62
|
```
|
59
63
|
|
60
|
-
Note: pass in 'xlist' if you would like sections to include cross listing information
|
64
|
+
*Note: pass in 'xlist' to your array of models if you would like sections to include cross listing information*
|
61
65
|
|
62
66
|
Example:
|
63
67
|
|
@@ -69,28 +73,8 @@ This will kick off a string of jobs to sync your specified models.
|
|
69
73
|
|
70
74
|
If you pass in the optional `term_scope` the provisioning reports will be run for only the terms returned by that scope. The scope must be defined on your `Term` model. (A sample one is provided in the generated `Term`.)
|
71
75
|
|
72
|
-
Imports are inserted in bulk with [https://github.com/zdennis/activerecord-import
|
73
|
-
|
74
|
-
## Legacy Support
|
75
|
-
|
76
|
-
If you have an old style tool that needs to sync data on a row by row basis, you can pass in the `legacy_support: true` option. In order for this to work, your models must have a `create_or_update_from_csv` class method defined that accepts a row argument. This method will get passed each row from the CSV, and it's up to you to persist it.
|
77
|
-
|
78
|
-
Example:
|
79
|
-
|
80
|
-
```ruby
|
81
|
-
CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active, legacy_support: true)
|
82
|
-
```
|
83
|
-
|
84
|
-
## CanvasSync::JobLog
|
85
|
-
|
86
|
-
Running the migrations will create a `canvas_sync_job_logs` table. All the jobs written in this gem will create a `CanvasSync::JobLog` and store data about their arguments, job class, any exceptions, and start/completion time. This will work regardless of your queue adapter.
|
87
|
-
|
88
|
-
If you want your own jobs to also log to the table all you have to do is have your job class inherit from `CanvasSync::Job`. You can also persist extra data you might need later by saving to the `metadata` column:
|
76
|
+
Imports are inserted in bulk with [activerecord-import](https://github.com/zdennis/activerecord-import) so they should be very fast.
|
89
77
|
|
90
|
-
```
|
91
|
-
@job_log.metadata = "This job ran really well!"
|
92
|
-
@job_log.save!
|
93
|
-
```
|
94
78
|
|
95
79
|
## Advanced Usage
|
96
80
|
|
@@ -125,7 +109,7 @@ class MyReallyCoolReportJob < CanvasSync::Jobs::ReportStarter
|
|
125
109
|
job_chain,
|
126
110
|
'my_really_cool_report_csv', # Report name
|
127
111
|
{ "parameters[param1]" => true }, # Report parameters
|
128
|
-
MyCoolProcessor, # Your processor class
|
112
|
+
MyCoolProcessor.to_s, # Your processor class as a string
|
129
113
|
options
|
130
114
|
)
|
131
115
|
end
|
@@ -144,7 +128,7 @@ The `CanvasSync.process_jobs` method allows you to pass in a chain of jobs to ru
|
|
144
128
|
{ job: JobClass, options: {} },
|
145
129
|
{ job: JobClass2, options: {} }
|
146
130
|
],
|
147
|
-
|
131
|
+
global_options: {}
|
148
132
|
}
|
149
133
|
```
|
150
134
|
|
@@ -156,7 +140,7 @@ job_chain = {
|
|
156
140
|
{ job: MyReallyCoolReportJob, options: {} },
|
157
141
|
{ job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
|
158
142
|
],
|
159
|
-
|
143
|
+
global_options: {}
|
160
144
|
}
|
161
145
|
|
162
146
|
CanvasSync.process_jobs(job_chain)
|
@@ -179,7 +163,7 @@ job_chain = {
|
|
179
163
|
{ job: SomeRandomJob, options: {} },
|
180
164
|
{ job: CanvasSync::Jobs::SyncProvisioningReportJob, options: { models: ['users', 'courses'] } }
|
181
165
|
],
|
182
|
-
|
166
|
+
global_options: {}
|
183
167
|
}
|
184
168
|
|
185
169
|
CanvasSync.process_jobs(job_chain)
|
@@ -187,7 +171,46 @@ CanvasSync.process_jobs(job_chain)
|
|
187
171
|
|
188
172
|
### Batching
|
189
173
|
|
190
|
-
The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable
|
174
|
+
The provisioning report uses the `CanvasSync::Importers::BulkImporter` class to bulk import rows with the activerecord-import gem. It inserts rows in batches of 10,000 by default. This can be customized by setting the `BULK_IMPORTER_BATCH_SIZE` environment variable.
|
175
|
+
|
176
|
+
### Mapping Overrides
|
177
|
+
|
178
|
+
Overrides are useful for two scenarios:
|
179
|
+
|
180
|
+
- You have an existing application where the column names do not match up with what CanvasSync expects
|
181
|
+
- You want to sync some other column in the report that CanvasSync is not configured to sync
|
182
|
+
|
183
|
+
In order to create an override, place a file called `canvas_sync_provisioning_mapping.yml` in your Rails `config` directory. Define the tables and columns you want to override using the following format:
|
184
|
+
|
185
|
+
```
|
186
|
+
users:
|
187
|
+
conflict_target: canvas_user_id # This must be a unique field that is present in the report and the database
|
188
|
+
report_columns: # The keys specified here are the column names in the report CSV
|
189
|
+
canvas_user_id_column_name_in_report:
|
190
|
+
database_column_name: canvas_user_id_name_in_your_db # Sometimes the database column name might not match the report column name
|
191
|
+
type: integer
|
192
|
+
```
|
193
|
+
|
194
|
+
## Legacy Support
|
195
|
+
|
196
|
+
If you have an old style tool that needs to sync data on a row by row basis, you can pass in the `legacy_support: true` option. In order for this to work, your models must have a `create_or_update_from_csv` class method defined that accepts a row argument. This method will get passed each row from the CSV, and it's up to you to persist it.
|
197
|
+
|
198
|
+
Example:
|
199
|
+
|
200
|
+
```ruby
|
201
|
+
CanvasSync.provisioning_sync(['users', 'courses'], term_scope: :active, legacy_support: true)
|
202
|
+
```
|
203
|
+
|
204
|
+
## CanvasSync::JobLog
|
205
|
+
|
206
|
+
Running the migrations will create a `canvas_sync_job_logs` table. All the jobs written in this gem will create a `CanvasSync::JobLog` and store data about their arguments, job class, any exceptions, and start/completion time. This will work regardless of your queue adapter.
|
207
|
+
|
208
|
+
If you want your own jobs to also log to the table all you have to do is have your job class inherit from `CanvasSync::Job`. You can also persist extra data you might need later by saving to the `metadata` column:
|
209
|
+
|
210
|
+
```
|
211
|
+
@job_log.metadata = "This job ran really well!"
|
212
|
+
@job_log.save!
|
213
|
+
```
|
191
214
|
|
192
215
|
## Upgrading
|
193
216
|
|
@@ -205,28 +228,16 @@ In order for this to work properly your database tables will need to have at lea
|
|
205
228
|
|
206
229
|
When adding to or updating this gem, make sure you do the following:
|
207
230
|
|
208
|
-
- If you modify a table that's already in the released version of the gem please write *new migrations* rather than modifying the existing ones. This will allow people to more easily upgrade.
|
209
231
|
- Update the yardoc comments where necessary, and confirm the changes by running `yardoc --server`
|
210
232
|
- Write specs
|
211
233
|
- If you modify the model or migration templates, run `bundle exec rake update_test_schema` to update them in the Rails Dummy application (and commit those changes)
|
212
234
|
|
235
|
+
## Docs
|
213
236
|
|
214
|
-
|
215
|
-
|
216
|
-
If your local database columns do not match up with those expected by CanvasSync you can define overrides.
|
217
|
-
In your Rails config folder, the canvas_sync_provisioning_mapping.yml file contains overrides for custom database column names
|
218
|
-
|
219
|
-
```
|
220
|
-
users:
|
221
|
-
conflict_target: canvas_user_id # Represents the database column that will determine if we need to update
|
222
|
-
report_columns: # The keys specified here are the column names in the report CSV
|
223
|
-
canvas_user_id_column_name_in_report:
|
224
|
-
database_column_name: canvas_user_id_name_in_your_db
|
225
|
-
type: integer
|
226
|
-
```
|
237
|
+
Docs can be generated using [yard](https://yardoc.org/). To view the docs:
|
227
238
|
|
228
|
-
|
239
|
+
- Clone this gem's repository
|
240
|
+
- `bundle install`
|
241
|
+
- `yard server --reload`
|
229
242
|
|
230
|
-
|
231
|
-
decently, but can be confusing. The difficulty is representing the options in a way that is easily serializable
|
232
|
-
by the queue adaptor and easily passed around.
|
243
|
+
The yard server will give you a URL you can visit to view the docs.
|
@@ -8,8 +8,8 @@ module CanvasSync
|
|
8
8
|
# Does a bulk import of a set of models using the activerecord-import gem.
|
9
9
|
#
|
10
10
|
# @param report_file_path [String] path to the report CSV
|
11
|
-
# @param mapping [Hash] a hash of the values to import.
|
12
|
-
#
|
11
|
+
# @param mapping [Hash] a hash of the values to import. See `model_mappings.yml` for a
|
12
|
+
# format example
|
13
13
|
# @param klass [Object] e.g., User
|
14
14
|
# @param conflict_target [Symbol] represents the database column that will determine if we need to update
|
15
15
|
# or insert a given row. e.g.,: canvas_user_id
|
data/lib/canvas_sync/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: canvas_sync
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nate Collings
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-11-
|
11
|
+
date: 2017-11-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|