rails_admin_import 1.3.1 → 2.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/CHANGELOG.md +30 -0
- data/README.md +251 -17
- data/Rakefile +3 -1
- data/app/views/rails_admin/main/import.html.haml +20 -6
- data/config/locales/README.md +3 -0
- data/config/locales/import.en.yml +47 -44
- data/lib/rails_admin_import/config.rb +6 -0
- data/lib/rails_admin_import/config/sections/import.rb +5 -1
- data/lib/rails_admin_import/formats/csv_importer.rb +9 -6
- data/lib/rails_admin_import/formats/json_importer.rb +1 -0
- data/lib/rails_admin_import/formats/xlsx_importer.rb +4 -2
- data/lib/rails_admin_import/import_logger.rb +1 -1
- data/lib/rails_admin_import/import_model.rb +12 -4
- data/lib/rails_admin_import/importer.rb +42 -14
- data/lib/rails_admin_import/version.rb +1 -1
- metadata +11 -26
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: b0b71f8452cd6e451b12a22c3d3afb9418bd9308857e11522007db476f014e29
|
4
|
+
data.tar.gz: df443cf15365e4c8ad61d81db2f9ba49bb582388cb9022f24ebeaacea2725b83
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 901d0f3cc189f9a51acf9e7cd492f66f28958a8a205c183f31138367caecb3639b7c6d47c66e866459e7b4739a2b51b93c44b47dc1c23f5d7932fe5563cffa8b
|
7
|
+
data.tar.gz: f9187fe631aa5e2e0ac0c2fcacc7d4a966d682fff9afefcb047608aaf4e9bb8587b56a615da95de7b429df05e9b9ecdff30c00f7320b0a0e3db090894036ba04
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,35 @@
|
|
1
1
|
# Change Log
|
2
2
|
|
3
|
+
# 2.3.0 / 2021-06-11
|
4
|
+
|
5
|
+
- Added an option to pass filename for each record. Thanks @waheedi
|
6
|
+
- Enumeration translation. Thanks @zaknafain
|
7
|
+
|
8
|
+
# 2.2.0 / 2019-02-24
|
9
|
+
|
10
|
+
- Switch to using CharlockHolmes for character detection. Thanks @codealchemy
|
11
|
+
- Readme improvements. Thanks @olleolleolle
|
12
|
+
|
13
|
+
# 2.1.0 / 2017-11-18
|
14
|
+
|
15
|
+
- French translation. Thanks @rodinux
|
16
|
+
- Check for blank headers. Thanks @JuandGirald
|
17
|
+
- Italian translation. Thanks @aprofiti
|
18
|
+
- Remove haml dependency. Thanks @prem-prakash
|
19
|
+
- Japanese translation. Thanks @higumachan
|
20
|
+
- Multiple mapping keys. Thanks @dmitrypol
|
21
|
+
- Add more hooks during import
|
22
|
+
|
23
|
+
# 2.0.0 / 2016-05-04
|
24
|
+
|
25
|
+
- Pull in the encoding list from the Encoding module instead of RailsAdmin. Thanks @baldursson, @patricklewis and @lucasff
|
26
|
+
|
27
|
+
# 1.4.0 / 2016-02-26
|
28
|
+
|
29
|
+
- Implement row limit (default maximum is 1000 rows). Thanks @codealchemy!
|
30
|
+
- Document mapping keys. Thanks @zrisher!
|
31
|
+
- Ignore columns with blank headers in CSV files
|
32
|
+
|
3
33
|
# 1.3.1 / 2016-02-11
|
4
34
|
|
5
35
|
- Bugfix: Create log/import directory when logging config option is enabled. Thanks @yovasx2!
|
data/README.md
CHANGED
@@ -1,17 +1,15 @@
|
|
1
1
|
# Rails Admin Import
|
2
2
|
|
3
|
-
[![Build Status](https://
|
3
|
+
[![Build Status](https://github.com/monkbroc/rails_admin_import/actions/workflows/ruby.yml/badge.svg)](https://github.com/monkbroc/rails_admin_import/actions/workflows/ruby.yml)
|
4
4
|
|
5
5
|
Plugin functionality to add generic import to Rails Admin from CSV, JSON and XLSX files
|
6
6
|
|
7
|
-
*This Readme is for version 1.x. If you are still using version 0.1.x, see [this branch](https://github.com/stephskardal/rails_admin_import/tree/legacy)*
|
8
|
-
|
9
7
|
## Installation
|
10
8
|
|
11
9
|
* First, add to Gemfile:
|
12
10
|
|
13
|
-
```
|
14
|
-
gem "rails_admin_import", "~>
|
11
|
+
```ruby
|
12
|
+
gem "rails_admin_import", "~> 2.3"
|
15
13
|
```
|
16
14
|
|
17
15
|
* Define configuration in `config/initializers/rails_admin_import.rb`:
|
@@ -28,8 +26,10 @@ RailsAdmin.config do |config|
|
|
28
26
|
|
29
27
|
# Optional:
|
30
28
|
# Configure global RailsAdminImport options
|
29
|
+
# Configure pass filename to records hashes
|
31
30
|
config.configure_with(:import) do |config|
|
32
31
|
config.logging = true
|
32
|
+
config.pass_filename = true
|
33
33
|
end
|
34
34
|
|
35
35
|
# Optional:
|
@@ -51,11 +51,109 @@ cannot :import, :all
|
|
51
51
|
can :import, [User, Model1, Model2]
|
52
52
|
```
|
53
53
|
|
54
|
-
##
|
54
|
+
## Usage
|
55
|
+
|
56
|
+
Model instances can be both created and updated from import data. Any fields
|
57
|
+
can be imported as long as they are allowed by the model's configuration.
|
58
|
+
Associated records can be looked up for both singular and plural relationships.
|
59
|
+
Both updating existing records and associating records requires the use of
|
60
|
+
**mapping keys**.
|
61
|
+
|
62
|
+
### Mapping Keys
|
63
|
+
|
64
|
+
Every importable class has a mapping key that uniquely identifies its
|
65
|
+
instances. The mapping key can be one or more fields. The value for
|
66
|
+
these fields can then be provided in import data, either to update the
|
67
|
+
existing record or to attach it through an association to another model.
|
68
|
+
This concept exists because `id`s are often not constant when moving
|
69
|
+
records between data stores.
|
70
|
+
|
71
|
+
For example, a `User` model may have an `email` field. When uploading a set
|
72
|
+
of users where some already exist in our database, we can select "email" as
|
73
|
+
our mapping key and then provide that field on each record in our data,
|
74
|
+
allowing us to update existing records with matching emails.
|
75
|
+
|
76
|
+
Using a csv formatted example:
|
77
|
+
|
78
|
+
```
|
79
|
+
Email,First name,Last name
|
80
|
+
peter.gibbons@initech.com,Peter,Gibbons
|
81
|
+
michael.bolton@initech.com,Michael,Bolton
|
82
|
+
```
|
83
|
+
|
84
|
+
would look for existing users with those emails. If one was found, its name
|
85
|
+
fields would be updated. Otherwise, a new one would be created.
|
86
|
+
|
87
|
+
For updating building owners, the mapping key could be `street_address` and
|
88
|
+
`zip_code`.
|
89
|
+
|
90
|
+
Similarly, if each user has favorite books, we could set the mapping key
|
91
|
+
for `Book` to be `isbn` and then include the isbn for their books within each
|
92
|
+
user record. The syntax for this is to use the name of the associated model as
|
93
|
+
the field name, no matter what actual mapping key has been selected. So
|
94
|
+
a user record would have one or more fields named "Book" that include each
|
95
|
+
associated book's ISBN.
|
96
|
+
|
97
|
+
Again using a csv formatted example:
|
98
|
+
|
99
|
+
```
|
100
|
+
Email, Book, Book, Book
|
101
|
+
peter.gibbons@initech.com, 9781119997870, 9780671027032
|
102
|
+
michael.bolton@initech.com, 9780446677479
|
103
|
+
```
|
104
|
+
|
105
|
+
would look up books with those ISBNs and attach them to those users.
|
106
|
+
|
107
|
+
Mapping keys can be selected on the import page. Their defaults can also be
|
108
|
+
globally configured in the config file:
|
109
|
+
|
110
|
+
```ruby
|
111
|
+
RailsAdmin.config do |config|
|
112
|
+
config.model 'User' do
|
113
|
+
import do
|
114
|
+
mapping_key :email
|
115
|
+
# for multiple values, use mapping_key [:first_name, :last_name]
|
116
|
+
mapping_key_list [:email, :some_other_id]
|
117
|
+
end
|
118
|
+
end
|
119
|
+
end
|
120
|
+
```
|
121
|
+
|
122
|
+
Since in models with large number of fields it doesn't make sense to use
|
123
|
+
most of them as mapping values, you can add `mapping_key_list` to
|
124
|
+
restrict which fields can be selected as mapping key in the UI during import.
|
125
|
+
|
126
|
+
Note that a matched record must exist when attaching associated models, or the
|
127
|
+
imported record will fail and be skipped.
|
128
|
+
|
129
|
+
Complex associations (`has_one ..., :through` or polymorphic associations)
|
130
|
+
need to be dealt with via custom logic called by one of the import hooks
|
131
|
+
(see below for more detail on using hooks). If we wanted to import
|
132
|
+
`Service`s and attach them to a `User`, but the user relationship
|
133
|
+
existed through an intermediary model called `ServiceProvider`, we could
|
134
|
+
provide a `user_email` field in our records and handle the actual
|
135
|
+
association with an import hook:
|
136
|
+
|
137
|
+
```ruby
|
138
|
+
class Service < ActiveRecord::Base
|
139
|
+
belongs_to :service_provider
|
140
|
+
has_one :user, through: :service_provider
|
141
|
+
|
142
|
+
def before_import_save(record)
|
143
|
+
if (email = record[:user_email]) && (user = User.find_by_email(email))
|
144
|
+
self.service_provider = user.service_provider
|
145
|
+
end
|
146
|
+
end
|
147
|
+
end
|
148
|
+
```
|
149
|
+
|
150
|
+
Importing new records by id is not recommended since it ignores the sequences of ids in database. That will lead to `ERROR: duplicate key value violates unique constraint` in future. You can work around this issue by adding an `import_id` column to your model, renaming the `id` column in your CSV to `import_id` and using `import_id` as the update lookup field.
|
151
|
+
|
152
|
+
### File format
|
55
153
|
|
56
154
|
The format is inferred by the extension (.csv, .json or .xlsx).
|
57
155
|
|
58
|
-
|
156
|
+
#### CSV
|
59
157
|
|
60
158
|
The first line must contain attribute names. They will be converted to lowercase and underscored (First Name ==> first_name).
|
61
159
|
|
@@ -71,29 +169,58 @@ Peter,Gibbons,IT,Management
|
|
71
169
|
Michael,Bolton,IT,
|
72
170
|
```
|
73
171
|
|
74
|
-
|
172
|
+
Blank lines will be skipped.
|
173
|
+
|
174
|
+
#### JSON
|
75
175
|
|
76
176
|
The file must be an array or an object with a root key the same name as the plural model name, i.e. the default Rails JSON output format with include_root_in_json on or off.
|
77
177
|
|
78
|
-
|
178
|
+
#### XLSX
|
79
179
|
|
80
180
|
The Microsoft Excel XLM format (XLSX) is supported, but not the old binary Microsoft Excel format (XLS).
|
81
181
|
|
82
182
|
The expected rows and columns are the same as for the CSV format (first line contains headers, multiple columns for "many" associations).
|
83
183
|
|
184
|
+
Blank lines will be skipped.
|
185
|
+
|
84
186
|
## Configuration
|
85
187
|
|
86
188
|
### Global configuration options
|
87
189
|
|
190
|
+
```ruby
|
191
|
+
RailsAdmin.config do |config|
|
192
|
+
config.actions do
|
193
|
+
all
|
194
|
+
import
|
195
|
+
end
|
196
|
+
|
197
|
+
# Default global RailsAdminImport options
|
198
|
+
config.configure_with(:import) do |config|
|
199
|
+
config.logging = false
|
200
|
+
config.line_item_limit = 1000
|
201
|
+
config.update_if_exists = false
|
202
|
+
config.pass_filename = false
|
203
|
+
config.rollback_on_error = false
|
204
|
+
config.header_converter = lambda do |header|
|
205
|
+
header.parameterize.underscore if header.present?
|
206
|
+
end
|
207
|
+
config.csv_options = {}
|
208
|
+
end
|
209
|
+
end
|
210
|
+
```
|
211
|
+
|
88
212
|
* __logging__ (default `false`): Save a copy of each imported file to log/import and a detailed import log to log/rails_admin_import.log
|
89
213
|
|
90
|
-
* __line_item_limit__ (default `1000`): max number of items that can be imported at one time.
|
214
|
+
* __line_item_limit__ (default `1000`): max number of items that can be imported at one time.
|
215
|
+
|
216
|
+
* __update_if_exists__ (default `false`): default value for the "Update if exists" checkbox on the import page.
|
91
217
|
|
92
218
|
* __rollback_on_error__ (default `false`): import records in a transaction and rollback if there is one error. Only for ActiveRecord, not Mongoid.
|
93
219
|
|
94
220
|
* __header_converter__ (default `lambda { ... }`): a lambda to convert each CSV header text string to a model attribute name. The default header converter converts to lowercase and replaces spaces with underscores.
|
95
221
|
|
96
222
|
* __csv_options__ (default `{}`): a hash of options that will be passed to a new [CSV](http://ruby-doc.org/stdlib-2.0.0/libdoc/csv/rdoc/CSV.html) instance
|
223
|
+
* __pass_filename__ (default `false`): Access the uploaded file name in your model actions, for example if set to true, inside each record, there will be an addtional property `record[:filename_importer]` which contains the file name of the currently uploaded file.
|
97
224
|
|
98
225
|
### Model-specific configuration
|
99
226
|
|
@@ -157,7 +284,7 @@ end
|
|
157
284
|
RailsAdmin.config do |config|
|
158
285
|
config.model 'User' do
|
159
286
|
import do
|
160
|
-
default_excluded_fields [:created_at, :updated_at]
|
287
|
+
default_excluded_fields [:created_at, :updated_at, :deleted_at, :c_at, :u_at]
|
161
288
|
end
|
162
289
|
end
|
163
290
|
end
|
@@ -165,18 +292,56 @@ end
|
|
165
292
|
|
166
293
|
## Import hooks
|
167
294
|
|
168
|
-
|
169
|
-
Define instance methods on your models to be hooked into the import process, if special/additional processing is required on the data:
|
295
|
+
Define methods on your models to be hooked into the import process, if special/additional processing is required on the data:
|
170
296
|
|
171
297
|
```ruby
|
172
298
|
# some model
|
173
299
|
class User < ActiveRecord::Base
|
300
|
+
def self.before_import
|
301
|
+
# called on the model class once before importing any individual records
|
302
|
+
end
|
303
|
+
|
304
|
+
def self.before_import_find(record)
|
305
|
+
# called on the model class before finding or creating the new record
|
306
|
+
# maybe modify the import record that will be used to find the model
|
307
|
+
# throw :skip to skip importing this record
|
308
|
+
throw :skip unless record[:email].ends_with? "@mycompany.com"
|
309
|
+
end
|
310
|
+
|
311
|
+
def before_import_attributes(record)
|
312
|
+
# called on the blank new model or the found model before fields are imported
|
313
|
+
# maybe delete fields from the import record that you don't need
|
314
|
+
# throw :skip to skip importing this record
|
315
|
+
end
|
316
|
+
|
317
|
+
def before_import_associations(record)
|
318
|
+
# called on the model with attributes but before associations are imported
|
319
|
+
# do custom import of associations
|
320
|
+
# make sure to delete association fields from the import record to avoid double import
|
321
|
+
record.delete(:my_association)
|
322
|
+
# throw :skip to skip importing this record
|
323
|
+
end
|
324
|
+
|
174
325
|
def before_import_save(record)
|
175
|
-
#
|
326
|
+
# called on the model before it is saved but after all fields and associations have been imported
|
327
|
+
# make final modifications to the record
|
328
|
+
# throw :skip to skip importing this record
|
176
329
|
end
|
177
330
|
|
178
331
|
def after_import_save(record)
|
179
|
-
#
|
332
|
+
# called on the model after it is saved
|
333
|
+
end
|
334
|
+
|
335
|
+
def after_import_association_error(record)
|
336
|
+
# called on the model when an association cannot be found
|
337
|
+
end
|
338
|
+
|
339
|
+
def after_import_error(record)
|
340
|
+
# called on the model when save fails
|
341
|
+
end
|
342
|
+
|
343
|
+
def self.after_import
|
344
|
+
# called once on the model class after importing all individual records
|
180
345
|
end
|
181
346
|
end
|
182
347
|
```
|
@@ -185,7 +350,35 @@ For example, you could
|
|
185
350
|
|
186
351
|
* Set an attribute on a Devise User model to skip checking for a password when importing a new model.
|
187
352
|
|
188
|
-
*
|
353
|
+
* Import an image into Carrierwave via a URL provided in the CSV.
|
354
|
+
|
355
|
+
```ruby
|
356
|
+
def before_import_save(record)
|
357
|
+
self.remote_image_url = record[:image] if record[:image].present?
|
358
|
+
end
|
359
|
+
```
|
360
|
+
|
361
|
+
* Skip some validations when importing.
|
362
|
+
|
363
|
+
```ruby
|
364
|
+
class User < ActiveRecord::Base
|
365
|
+
# Non-persistent attribute to allow creating a new user without a password
|
366
|
+
# Password will be set by the user by following a link in the invitation email
|
367
|
+
attr_accessor :allow_blank_password
|
368
|
+
|
369
|
+
devise :validatable
|
370
|
+
|
371
|
+
# Called by Devise to enable/disable password presence validation
|
372
|
+
def password_required?
|
373
|
+
allow_blank_password ? false : super
|
374
|
+
end
|
375
|
+
|
376
|
+
# Don't require a password when importing users
|
377
|
+
def before_import_save(record)
|
378
|
+
self.allow_blank_password = true
|
379
|
+
end
|
380
|
+
end
|
381
|
+
```
|
189
382
|
|
190
383
|
|
191
384
|
## ORM: ActiveRecord and Mongoid
|
@@ -201,10 +394,32 @@ Since the import functionality is rarely used in many applications, some gems ar
|
|
201
394
|
|
202
395
|
If you prefer to eager load all dependecies at boot, use this line in your `Gemfile`.
|
203
396
|
|
204
|
-
```
|
397
|
+
```ruby
|
205
398
|
gem "rails_admin_import", "~> 1.2.0", require: "rails_admin_import/eager_load"
|
206
399
|
```
|
207
400
|
|
401
|
+
## Import error due to Rails class reloading
|
402
|
+
|
403
|
+
![error due to class reloading](https://user-images.githubusercontent.com/2566348/51355874-0f83ad00-1a7e-11e9-8e58-46bc4699f2e6.jpg)
|
404
|
+
|
405
|
+
If you get an error like `Error during import: MyModel(#70286054976500) expected, got MyModel(#70286114743280)`, you need restart the rails server and redo the import. This is due to the fact that Rails reloads the ActiveRecord model classes in development when you make changes to them and Rails Admin is still using the old class.
|
406
|
+
|
407
|
+
Another suggestion is to set `config.cache_classes = true` to true in your `development.rb` for Rails Admin Import to work around the ActiveRecord model class reloading issue. See [this comment](https://github.com/stephskardal/rails_admin_import/issues/88#issuecomment-455374671) for more information.
|
408
|
+
|
409
|
+
## Customize the UI
|
410
|
+
|
411
|
+
If you want to hide all the advanced fields from the import UI, you can copy [`app/views/rails_admin/main/import.html.haml`](app/views/rails_admin/main/import.html.haml) to your project at the same path. Add `.hidden` at the end of lines you want to hide.
|
412
|
+
|
413
|
+
For example:
|
414
|
+
|
415
|
+
```haml
|
416
|
+
.form-group.control-group.hidden
|
417
|
+
%label.col-sm-2.control-label= t("admin.import.update_if_exists")
|
418
|
+
.col-sm-10.controls
|
419
|
+
= check_box_tag :update_if_exists, '1', true, :class => "form-control"
|
420
|
+
%p.help-block= t('admin.import.help.update_if_exists')
|
421
|
+
```
|
422
|
+
|
208
423
|
## Upgrading
|
209
424
|
|
210
425
|
* Move global config to `config.configure_with(:import)` in `config/initializers/rails_admin_import.rb`.
|
@@ -217,6 +432,18 @@ gem "rails_admin_import", "~> 1.2.0", require: "rails_admin_import/eager_load"
|
|
217
432
|
|
218
433
|
* Support for importing file attributes was removed since I couldn't understand how it works. It should be possible to reimplement it yourself using post import hooks. Open an issue to discuss how to put back support for importing files into the gem.
|
219
434
|
|
435
|
+
## Community-contributed translations
|
436
|
+
|
437
|
+
* [Spanish translation](https://gist.github.com/yovasx2/dc0e9512e6c6243f840c) by Giovanni Alberto
|
438
|
+
|
439
|
+
* [French translation](https://github.com/rodinux/rails_admin_import.fr-MX.yml) by Rodolphe Robles. (I suggest to translate also rails admin.fr and your locales.fr to resolve an issue with DatePicker)
|
440
|
+
|
441
|
+
* [Italian translation](https://gist.github.com/aprofiti/ec3dc452898c8c48534b59eeb2701765) by Alessandro Profiti
|
442
|
+
|
443
|
+
* [Japanese translation](https://gist.github.com/higumachan/c4bf669d6446ec509386229f916ba5fc) by Yuta Hinokuma
|
444
|
+
|
445
|
+
* [Brazilian Portuguese translation](https://gist.github.com/tteurs/5a87ff4bc5f24692dab05b3cde0ca9df) by Matheo Gracia Pegoraro
|
446
|
+
|
220
447
|
## Run tests
|
221
448
|
|
222
449
|
1. Clone the repository to your machine
|
@@ -235,6 +462,13 @@ Original author: [Steph Skardal](https://github.com/stephskardal)
|
|
235
462
|
Maintainer (since May 2015): [Julien Vanier](https://github.com/monkbroc)
|
236
463
|
|
237
464
|
|
465
|
+
## Release
|
466
|
+
|
467
|
+
- Update `lib/rails_admin_import/version.rb`
|
468
|
+
- Update the install instructions at [the top of the readme](#installation)
|
469
|
+
- Commit to git
|
470
|
+
- `rake release`
|
471
|
+
|
238
472
|
## Contributing
|
239
473
|
|
240
474
|
Everyone is encouraged to help improve this project. Here are a few ways you can help:
|
data/Rakefile
CHANGED
@@ -1,3 +1,14 @@
|
|
1
|
+
:ruby
|
2
|
+
translations = {
|
3
|
+
add: t('admin.import.enumeration.add'),
|
4
|
+
chooseAll: t('admin.import.enumeration.choose_all'),
|
5
|
+
clearAll: t('admin.import.enumeration.clear_all'),
|
6
|
+
down: t('admin.import.enumeration.down'),
|
7
|
+
remove: t('admin.import.enumeration.remove'),
|
8
|
+
search: t('admin.import.enumeration.search'),
|
9
|
+
up: t('admin.import.enumeration.up')
|
10
|
+
}
|
11
|
+
|
1
12
|
= render "results"
|
2
13
|
|
3
14
|
= form_tag import_path(@abstract_model), :multipart => true, class: 'form-horizontal denser' do
|
@@ -24,20 +35,22 @@
|
|
24
35
|
%label.col-sm-2.control-label{for: "encoding"}= t("admin.import.encoding")
|
25
36
|
.col-sm-10.controls
|
26
37
|
= select_tag 'encoding',
|
27
|
-
options_for_select(
|
28
|
-
include_blank: true, data: { enumeration: true }
|
38
|
+
options_for_select(Encoding.name_list.sort),
|
39
|
+
include_blank: true, data: { enumeration: true, options: { regional: translations } }
|
29
40
|
%p.help-block= t('admin.import.help.encoding', name: 'UTF-8')
|
30
41
|
.form-group.control-group
|
31
42
|
%label.col-sm-2.control-label= t("admin.import.update_if_exists")
|
32
43
|
.col-sm-10.controls
|
33
|
-
= check_box_tag :update_if_exists, '1',
|
44
|
+
= check_box_tag :update_if_exists, '1', RailsAdminImport.config.update_if_exists, :class => "form-control"
|
34
45
|
%p.help-block= t('admin.import.help.update_if_exists')
|
35
46
|
.form-group.control-group
|
36
47
|
%label.col-sm-2.control-label{for: "update_lookup"}= t("admin.import.update_lookup")
|
37
48
|
.col-sm-10.controls
|
38
49
|
= select_tag 'update_lookup',
|
39
50
|
options_for_select(@import_model.update_lookup_field_names,
|
40
|
-
|
51
|
+
Array.wrap(@import_model.config.mapping_key).map(&:to_s)),
|
52
|
+
multiple: true,
|
53
|
+
data: { enumeration: true, options: { regional: translations } }
|
41
54
|
|
42
55
|
- unless @import_model.association_fields.empty?
|
43
56
|
%fieldset
|
@@ -52,8 +65,9 @@
|
|
52
65
|
= t("admin.import.mapping")
|
53
66
|
.col-sm-10.controls
|
54
67
|
= select_tag "associations[#{field.name}]",
|
55
|
-
options_for_select(@import_model.associated_model_fields(field)
|
56
|
-
|
68
|
+
options_for_select(@import_model.associated_model_fields(field),
|
69
|
+
Array.wrap(@import_model.associated_config(field).mapping_key).first.to_s),
|
70
|
+
data: { enumeration: true, options: { regional: translations } }
|
57
71
|
|
58
72
|
%br
|
59
73
|
.form-actions
|
@@ -0,0 +1,3 @@
|
|
1
|
+
See the [community contributed translation section of the README](https://github.com/stephskardal/rails_admin_import#community-contributed-translations) for more languages.
|
2
|
+
|
3
|
+
To contribute a new translation, please put it in a gist and submit a pull request to link your translation in the README.
|
@@ -1,50 +1,53 @@
|
|
1
|
-
|
2
1
|
en:
|
3
2
|
admin:
|
4
3
|
actions:
|
5
4
|
import:
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
link:
|
10
|
-
|
11
|
-
|
5
|
+
breadcrumb: Import
|
6
|
+
bulk_link: Import
|
7
|
+
done: Imported
|
8
|
+
link: Import
|
9
|
+
menu: Import
|
10
|
+
title: Import
|
12
11
|
import:
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
legend:
|
27
|
-
fields: "Fields to import"
|
28
|
-
upload: "Upload file"
|
29
|
-
mapping: "Related fields mapping"
|
30
|
-
import_success:
|
31
|
-
create: "Created %{name}"
|
32
|
-
update: "Updated %{name}"
|
33
|
-
import_error:
|
34
|
-
create: "Failed to create %{name}: %{error}"
|
35
|
-
update: "Failed to update %{name}: %{error}"
|
36
|
-
general: "Error during import: %{error}"
|
37
|
-
old_import_hook: >
|
38
|
-
The import hook %{model}.%{method} should take only 1 argument.
|
39
|
-
Data may not imported correctly.
|
40
|
-
See Upgrading section readme in Rails Admin Import.
|
41
|
-
association_not_found: "Association not found. %{error}"
|
12
|
+
association_fields: Association fields
|
13
|
+
association_not_found: 'Association not found. %{error}'
|
14
|
+
encoding: Encoding
|
15
|
+
enumeration:
|
16
|
+
add: Add new
|
17
|
+
choose_all: Choose all
|
18
|
+
clear_all: Clear all
|
19
|
+
down: Down
|
20
|
+
remove: Remove
|
21
|
+
search: Search
|
22
|
+
up: Up
|
23
|
+
file: Data file
|
24
|
+
format: File format
|
42
25
|
help:
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
26
|
+
association_fields: |
|
27
|
+
These fields map to other tables in the database, lookup via attribute selected below. For "many" associations, you may include multiple columns with the same header in the CSV file.
|
28
|
+
encoding: Choose file encoding. Leave empty to auto-detect. Ignored for JSON.
|
29
|
+
file_limit: 'Please limit upload file to %{limit} line items.'
|
30
|
+
model_fields: The fields above may be included in the import file.
|
31
|
+
update_if_exists: Update records found with the lookup field below instead of creating new records
|
32
|
+
import_error:
|
33
|
+
create: 'Failed to create %{name}: %{error}'
|
34
|
+
general: 'Error during import: %{error}'
|
35
|
+
line_item_limit: 'Please limit upload file to %{limit} line items.'
|
36
|
+
old_import_hook: |
|
37
|
+
The import hook %{model}.%{method} should take only 1 argument. Data may not imported correctly. See Upgrading section readme in Rails Admin Import.
|
38
|
+
update: 'Failed to update %{name}: %{error}'
|
39
|
+
import_success:
|
40
|
+
create: 'Created %{name}'
|
41
|
+
update: 'Updated %{name}'
|
42
|
+
invalid_format: Invalid import format.
|
43
|
+
invalid_json: 'The JSON data should be an array of records or an object with a key ''%{root_key}'' set to an array of records'
|
44
|
+
legend:
|
45
|
+
fields: Fields to import
|
46
|
+
mapping: Related fields mapping
|
47
|
+
upload: Upload file
|
48
|
+
mapping: mapping
|
49
|
+
missing_file: You must select a file
|
50
|
+
missing_update_lookup: Your file must contain a column for the 'Update lookup field' you selected.
|
51
|
+
model_fields: Model fields
|
52
|
+
update_if_exists: Update if exists
|
53
|
+
update_lookup: Update lookup field(s)
|
@@ -6,11 +6,15 @@ module RailsAdminImport
|
|
6
6
|
attr_accessor :logging
|
7
7
|
attr_accessor :line_item_limit
|
8
8
|
attr_accessor :rollback_on_error
|
9
|
+
attr_accessor :update_if_exists
|
9
10
|
attr_accessor :header_converter
|
10
11
|
attr_accessor :csv_options
|
12
|
+
attr_accessor :pass_filename
|
11
13
|
|
12
14
|
# Default is to downcase headers and add underscores to convert into attribute names
|
13
15
|
HEADER_CONVERTER = lambda do |header|
|
16
|
+
# check for nil/blank headers
|
17
|
+
next if header.blank?
|
14
18
|
header.parameterize.underscore
|
15
19
|
end
|
16
20
|
|
@@ -31,7 +35,9 @@ module RailsAdminImport
|
|
31
35
|
@logging = false
|
32
36
|
@line_item_limit = 1000
|
33
37
|
@rollback_on_error = false
|
38
|
+
@update_if_exists = false
|
34
39
|
@header_converter = HEADER_CONVERTER
|
40
|
+
@pass_filename = false
|
35
41
|
@csv_options = {}
|
36
42
|
end
|
37
43
|
end
|
@@ -9,8 +9,12 @@ module RailsAdmin
|
|
9
9
|
:name
|
10
10
|
end
|
11
11
|
|
12
|
+
register_instance_option(:mapping_key_list) do
|
13
|
+
[]
|
14
|
+
end
|
15
|
+
|
12
16
|
register_instance_option(:default_excluded_fields) do
|
13
|
-
[:id, :_id, :created_at, :updated_at]
|
17
|
+
[:id, :_id, :created_at, :updated_at, :c_at, :u_at, :deleted_at]
|
14
18
|
end
|
15
19
|
end
|
16
20
|
end
|
@@ -1,9 +1,11 @@
|
|
1
1
|
require "csv"
|
2
|
+
require "charlock_holmes"
|
2
3
|
|
3
4
|
module RailsAdminImport
|
4
5
|
module Formats
|
5
6
|
class CSVImporter < FileImporter
|
6
7
|
Formats.register(:csv, self)
|
8
|
+
Formats.register(:CSV, self)
|
7
9
|
|
8
10
|
autoload :CharDet, "rchardet"
|
9
11
|
|
@@ -15,8 +17,9 @@ module RailsAdminImport
|
|
15
17
|
|
16
18
|
# A method that yields a hash of attributes for each record to import
|
17
19
|
def each_record
|
18
|
-
CSV.foreach(filename, csv_options) do |row|
|
19
|
-
|
20
|
+
CSV.foreach(filename, **csv_options) do |row|
|
21
|
+
attr = convert_to_attributes(row)
|
22
|
+
yield attr unless attr.all? { |field, value| value.blank? }
|
20
23
|
end
|
21
24
|
end
|
22
25
|
|
@@ -51,9 +54,9 @@ module RailsAdminImport
|
|
51
54
|
end
|
52
55
|
|
53
56
|
def detect_encoding
|
54
|
-
charset =
|
55
|
-
if charset[
|
56
|
-
from_encoding = charset[
|
57
|
+
charset = CharlockHolmes::EncodingDetector.detect File.read(filename)
|
58
|
+
if charset[:confidence] > 0.6
|
59
|
+
from_encoding = charset[:encoding]
|
57
60
|
from_encoding = "UTF-8" if from_encoding == "ascii"
|
58
61
|
end
|
59
62
|
from_encoding
|
@@ -61,7 +64,7 @@ module RailsAdminImport
|
|
61
64
|
|
62
65
|
def convert_to_attributes(row)
|
63
66
|
row.each_with_object({}) do |(field, value), record|
|
64
|
-
|
67
|
+
next if field.blank?
|
65
68
|
field = field.to_sym
|
66
69
|
if import_model.has_multiple_values?(field)
|
67
70
|
field = import_model.pluralize_field(field)
|
@@ -4,6 +4,7 @@ module RailsAdminImport
|
|
4
4
|
module Formats
|
5
5
|
class XLSXImporter < FileImporter
|
6
6
|
Formats.register(:xlsx, self)
|
7
|
+
Formats.register(:XLSX, self)
|
7
8
|
|
8
9
|
autoload :SimpleXlsxReader, "simple_xlsx_reader"
|
9
10
|
|
@@ -18,7 +19,8 @@ module RailsAdminImport
|
|
18
19
|
sheet = doc.sheets.first
|
19
20
|
@headers = convert_headers(sheet.headers)
|
20
21
|
sheet.data.each do |row|
|
21
|
-
|
22
|
+
attr = convert_to_attributes(row)
|
23
|
+
yield attr unless attr.all? { |field, value| value.blank? }
|
22
24
|
end
|
23
25
|
end
|
24
26
|
|
@@ -33,7 +35,7 @@ module RailsAdminImport
|
|
33
35
|
def convert_to_attributes(row)
|
34
36
|
row_with_headers = @headers.zip(row)
|
35
37
|
row_with_headers.each_with_object({}) do |(field, value), record|
|
36
|
-
|
38
|
+
next if field.nil?
|
37
39
|
field = field.to_sym
|
38
40
|
if import_model.has_multiple_values?(field)
|
39
41
|
field = import_model.pluralize_field(field)
|
@@ -63,7 +63,11 @@ module RailsAdminImport
|
|
63
63
|
end
|
64
64
|
|
65
65
|
def update_lookup_field_names
|
66
|
-
@
|
66
|
+
if @config.mapping_key_list.present?
|
67
|
+
@update_lookup_field_names = @config.mapping_key_list
|
68
|
+
else
|
69
|
+
@update_lookup_field_names ||= model_fields.map(&:name) + belongs_to_fields.map(&:foreign_key)
|
70
|
+
end
|
67
71
|
end
|
68
72
|
|
69
73
|
def associated_object(field, mapping_field, value)
|
@@ -82,9 +86,13 @@ module RailsAdminImport
|
|
82
86
|
|
83
87
|
def associated_model_fields(field)
|
84
88
|
@associated_fields ||= {}
|
85
|
-
|
86
|
-
|
87
|
-
|
89
|
+
if associated_config(field).mapping_key_list.present?
|
90
|
+
@associated_fields[field] ||= associated_config(field).mapping_key_list
|
91
|
+
else
|
92
|
+
@associated_fields[field] ||= associated_config(field).visible_fields.select { |f|
|
93
|
+
!f.association?
|
94
|
+
}.map(&:name)
|
95
|
+
end
|
88
96
|
end
|
89
97
|
|
90
98
|
def has_multiple_values?(field_name)
|
@@ -16,18 +16,26 @@ module RailsAdminImport
|
|
16
16
|
begin
|
17
17
|
init_results
|
18
18
|
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
19
|
+
if records.count > RailsAdminImport.config.line_item_limit
|
20
|
+
return results = {
|
21
|
+
success: [],
|
22
|
+
error: [I18n.t('admin.import.import_error.line_item_limit', limit: RailsAdminImport.config.line_item_limit)]
|
23
|
+
}
|
24
|
+
end
|
25
|
+
|
26
|
+
perform_global_callback(:before_import)
|
23
27
|
|
24
28
|
with_transaction do
|
25
29
|
records.each do |record|
|
26
|
-
|
30
|
+
catch :skip do
|
31
|
+
import_record(record)
|
32
|
+
end
|
27
33
|
end
|
28
34
|
|
29
35
|
rollback_if_error
|
30
36
|
end
|
37
|
+
|
38
|
+
perform_global_callback(:after_import)
|
31
39
|
rescue Exception => e
|
32
40
|
report_general_error("#{e} (#{e.backtrace.first})")
|
33
41
|
end
|
@@ -62,19 +70,28 @@ module RailsAdminImport
|
|
62
70
|
end
|
63
71
|
|
64
72
|
def import_record(record)
|
65
|
-
if
|
73
|
+
if params["file"] && RailsAdminImport.config.pass_filename
|
74
|
+
record.merge!({:filename_importer => params[:file].original_filename})
|
75
|
+
end
|
76
|
+
|
77
|
+
perform_model_callback(import_model.model, :before_import_find, record)
|
78
|
+
|
79
|
+
if update_lookup && !(update_lookup - record.keys).empty?
|
66
80
|
raise UpdateLookupError, I18n.t("admin.import.missing_update_lookup")
|
67
|
-
end
|
81
|
+
end
|
68
82
|
|
69
83
|
object = find_or_create_object(record, update_lookup)
|
84
|
+
return if object.nil?
|
70
85
|
action = object.new_record? ? :create : :update
|
71
86
|
|
72
87
|
begin
|
88
|
+
perform_model_callback(object, :before_import_associations, record)
|
73
89
|
import_single_association_data(object, record)
|
74
90
|
import_many_association_data(object, record)
|
75
91
|
rescue AssociationNotFound => e
|
76
92
|
error = I18n.t("admin.import.association_not_found", :error => e.to_s)
|
77
93
|
report_error(object, action, error)
|
94
|
+
perform_model_callback(object, :after_import_association_error, record)
|
78
95
|
return
|
79
96
|
end
|
80
97
|
|
@@ -85,12 +102,13 @@ module RailsAdminImport
|
|
85
102
|
perform_model_callback(object, :after_import_save, record)
|
86
103
|
else
|
87
104
|
report_error(object, action, object.errors.full_messages.join(", "))
|
105
|
+
perform_model_callback(object, :after_import_error, record)
|
88
106
|
end
|
89
107
|
end
|
90
108
|
|
91
109
|
def update_lookup
|
92
110
|
@update_lookup ||= if params[:update_if_exists] == "1"
|
93
|
-
params[:update_lookup].to_sym
|
111
|
+
params[:update_lookup].map(&:to_sym)
|
94
112
|
end
|
95
113
|
end
|
96
114
|
|
@@ -142,7 +160,7 @@ module RailsAdminImport
|
|
142
160
|
name: result_count,
|
143
161
|
action: I18n.t("admin.actions.import.done"))
|
144
162
|
end
|
145
|
-
|
163
|
+
|
146
164
|
def perform_model_callback(object, method_name, record)
|
147
165
|
if object.respond_to?(method_name)
|
148
166
|
# Compatibility: Old import hook took 2 arguments.
|
@@ -166,6 +184,11 @@ module RailsAdminImport
|
|
166
184
|
end
|
167
185
|
end
|
168
186
|
|
187
|
+
def perform_global_callback(method_name)
|
188
|
+
object = import_model.model
|
189
|
+
object.send(method_name) if object.respond_to?(method_name)
|
190
|
+
end
|
191
|
+
|
169
192
|
def find_or_create_object(record, update)
|
170
193
|
field_names = import_model.model_fields.map(&:name)
|
171
194
|
new_attrs = record.select do |field_name, value|
|
@@ -174,13 +197,19 @@ module RailsAdminImport
|
|
174
197
|
|
175
198
|
model = import_model.model
|
176
199
|
object = if update.present?
|
177
|
-
|
178
|
-
|
200
|
+
query = update.each_with_object({}) do
|
201
|
+
|field, query| query[field] = record[field]
|
202
|
+
end
|
203
|
+
model.where(query).first
|
204
|
+
end
|
179
205
|
|
180
206
|
if object.nil?
|
181
|
-
object = model.new
|
207
|
+
object = model.new
|
208
|
+
perform_model_callback(object, :before_import_attributes, record)
|
209
|
+
object.attributes = new_attrs
|
182
210
|
else
|
183
|
-
object
|
211
|
+
perform_model_callback(object, :before_import_attributes, record)
|
212
|
+
object.attributes = new_attrs.except(update.map(&:to_sym))
|
184
213
|
end
|
185
214
|
object
|
186
215
|
end
|
@@ -221,4 +250,3 @@ module RailsAdminImport
|
|
221
250
|
end
|
222
251
|
end
|
223
252
|
end
|
224
|
-
|
metadata
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rails_admin_import
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.3.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Steph Skardal
|
8
8
|
- Julien Vanier
|
9
|
-
autorequire:
|
9
|
+
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2021-06-12 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: rails
|
@@ -40,33 +40,19 @@ dependencies:
|
|
40
40
|
- !ruby/object:Gem::Version
|
41
41
|
version: 0.6.6
|
42
42
|
- !ruby/object:Gem::Dependency
|
43
|
-
name:
|
43
|
+
name: charlock_holmes
|
44
44
|
requirement: !ruby/object:Gem::Requirement
|
45
45
|
requirements:
|
46
46
|
- - "~>"
|
47
47
|
- !ruby/object:Gem::Version
|
48
|
-
version: '
|
48
|
+
version: '0.7'
|
49
49
|
type: :runtime
|
50
50
|
prerelease: false
|
51
51
|
version_requirements: !ruby/object:Gem::Requirement
|
52
52
|
requirements:
|
53
53
|
- - "~>"
|
54
54
|
- !ruby/object:Gem::Version
|
55
|
-
version: '
|
56
|
-
- !ruby/object:Gem::Dependency
|
57
|
-
name: rchardet
|
58
|
-
requirement: !ruby/object:Gem::Requirement
|
59
|
-
requirements:
|
60
|
-
- - "~>"
|
61
|
-
- !ruby/object:Gem::Version
|
62
|
-
version: '1.6'
|
63
|
-
type: :runtime
|
64
|
-
prerelease: false
|
65
|
-
version_requirements: !ruby/object:Gem::Requirement
|
66
|
-
requirements:
|
67
|
-
- - "~>"
|
68
|
-
- !ruby/object:Gem::Version
|
69
|
-
version: '1.6'
|
55
|
+
version: '0.7'
|
70
56
|
- !ruby/object:Gem::Dependency
|
71
57
|
name: simple_xlsx_reader
|
72
58
|
requirement: !ruby/object:Gem::Requirement
|
@@ -81,7 +67,7 @@ dependencies:
|
|
81
67
|
- - "~>"
|
82
68
|
- !ruby/object:Gem::Version
|
83
69
|
version: '1.0'
|
84
|
-
description:
|
70
|
+
description:
|
85
71
|
email:
|
86
72
|
- steph@endpoint.com
|
87
73
|
- jvanier@gmail.com
|
@@ -96,6 +82,7 @@ files:
|
|
96
82
|
- app/views/rails_admin/main/_results.html.haml
|
97
83
|
- app/views/rails_admin/main/_section.html.haml
|
98
84
|
- app/views/rails_admin/main/import.html.haml
|
85
|
+
- config/locales/README.md
|
99
86
|
- config/locales/import.en.yml
|
100
87
|
- lib/rails_admin_import.rb
|
101
88
|
- lib/rails_admin_import/action.rb
|
@@ -119,7 +106,7 @@ homepage: https://github.com/stephskardal/rails_admin_import
|
|
119
106
|
licenses:
|
120
107
|
- MIT
|
121
108
|
metadata: {}
|
122
|
-
post_install_message:
|
109
|
+
post_install_message:
|
123
110
|
rdoc_options: []
|
124
111
|
require_paths:
|
125
112
|
- lib
|
@@ -134,10 +121,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
134
121
|
- !ruby/object:Gem::Version
|
135
122
|
version: '0'
|
136
123
|
requirements: []
|
137
|
-
|
138
|
-
|
139
|
-
signing_key:
|
124
|
+
rubygems_version: 3.2.15
|
125
|
+
signing_key:
|
140
126
|
specification_version: 4
|
141
127
|
summary: Import functionality for Rails Admin
|
142
128
|
test_files: []
|
143
|
-
has_rdoc:
|