bulkrax 5.4.2 → 5.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +13 -6
- data/app/assets/javascripts/bulkrax/exporters.js +5 -5
- data/app/assets/javascripts/bulkrax/navtabs.js.erb +19 -2
- data/app/factories/bulkrax/object_factory.rb +2 -2
- data/app/jobs/bulkrax/create_relationships_job.rb +5 -4
- data/app/jobs/bulkrax/delete_job.rb +5 -5
- data/app/jobs/bulkrax/download_cloud_file_job.rb +1 -1
- data/app/jobs/bulkrax/export_work_job.rb +6 -6
- data/app/jobs/bulkrax/import_collection_job.rb +7 -7
- data/app/jobs/bulkrax/import_file_set_job.rb +7 -7
- data/app/jobs/bulkrax/import_work_job.rb +6 -6
- data/app/jobs/bulkrax/importer_job.rb +1 -1
- data/app/models/bulkrax/csv_collection_entry.rb +1 -1
- data/app/models/bulkrax/csv_entry.rb +4 -2
- data/app/parsers/bulkrax/application_parser.rb +2 -2
- data/app/parsers/bulkrax/csv_parser.rb +3 -2
- data/app/parsers/bulkrax/oai_dc_parser.rb +1 -1
- data/app/services/bulkrax/remove_relationships_for_importer.rb +2 -1
- data/app/views/bulkrax/exporters/_form.html.erb +10 -10
- data/app/views/bulkrax/exporters/show.html.erb +3 -3
- data/app/views/bulkrax/importers/show.html.erb +3 -3
- data/app/views/bulkrax/shared/_bulkrax_errors.html.erb +2 -2
- data/app/views/hyrax/dashboard/sidebar/_bulkrax_sidebar_additions.html.erb +12 -4
- data/db/migrate/20190731114016_change_importer_and_exporter_to_polymorphic.rb +6 -4
- data/lib/bulkrax/entry_spec_helper.rb +50 -26
- data/lib/bulkrax/version.rb +1 -1
- data/lib/bulkrax.rb +198 -207
- data/lib/generators/bulkrax/templates/config/initializers/bulkrax.rb +1 -0
- metadata +2 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5935f574678e9ff331fb1c3e637512e81aaf133acd44ac6ba413828b663c3cf1
|
4
|
+
data.tar.gz: d4a44f4fe47171d22c131bb836ec348091c88fcaa011a2e3a61b4234c8bb3be3
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d14475603489ad27d29db6536e4e6d50e3d3393845724a7d85cf0c674804f3b42fbe1430ff9eea0d7e16d5f0482760ca1615e5a55b66a791ff5eab60f1950ea6
|
7
|
+
data.tar.gz: e593a46744c543b2d9e140010178272b470427f726d3f91504b2e6969a15632f5cf467abe375f588b10b2b9821bcdc3b23730ecccb071b5e0411bbc4e794eb9c
|
data/README.md
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
![Test Suite](https://github.com/samvera/bulkrax/actions/workflows/test.yml/badge.svg)
|
2
|
+
![Test Suite](https://github.com/samvera/bulkrax/actions/workflows/lint.yml/badge.svg)
|
1
3
|
# Bulkrax
|
2
4
|
Bulkrax is a batteries included importer for Samvera applications. It currently includes support for OAI-PMH (DC and Qualified DC) and CSV out of the box. It is also designed to be extensible, allowing you to easily add new importers in to your application or to include them with other gems. Bulkrax provides a full admin interface including creating, editing, scheduling and reviewing imports.
|
3
5
|
|
@@ -10,7 +12,7 @@ Add this line to your application's Gemfile:
|
|
10
12
|
```ruby
|
11
13
|
gem 'bulkrax'
|
12
14
|
# or if using from github
|
13
|
-
gem 'bulkrax', git: 'https://github.com/samvera
|
15
|
+
gem 'bulkrax', git: 'https://github.com/samvera/bulkrax.git', branch: 'main'
|
14
16
|
```
|
15
17
|
|
16
18
|
And then execute:
|
@@ -72,7 +74,7 @@ Bulkrax.setup do |config|
|
|
72
74
|
end
|
73
75
|
```
|
74
76
|
|
75
|
-
The [configuration guide](https://github.com/samvera
|
77
|
+
The [configuration guide](https://github.com/samvera/bulkrax/wiki/Configuring-Bulkrax) provides detailed instructions on the various available configurations.
|
76
78
|
|
77
79
|
Example:
|
78
80
|
|
@@ -122,7 +124,7 @@ It's unlikely that the incoming import data has fields that exactly match those
|
|
122
124
|
|
123
125
|
By default, a mapping for the OAI parser has been added to map standard oai_dc fields to Hyrax basic_metadata. The other parsers have no default mapping, and will map any incoming fields to Hyrax properties with the same name. Configurations can be added in `config/initializers/bulkrax.rb`
|
124
126
|
|
125
|
-
Configuring field mappings is documented in the [Bulkrax Configuration Guide](https://github.com/samvera
|
127
|
+
Configuring field mappings is documented in the [Bulkrax Configuration Guide](https://github.com/samvera/bulkrax/wiki/Configuring-Bulkrax).
|
126
128
|
|
127
129
|
## Importing Files
|
128
130
|
|
@@ -153,7 +155,7 @@ end
|
|
153
155
|
|
154
156
|
## Customizing Bulkrax
|
155
157
|
|
156
|
-
For further information on how to extend and customize Bulkrax, please see the [Bulkrax Customization Guide](https://github.com/samvera
|
158
|
+
For further information on how to extend and customize Bulkrax, please see the [Bulkrax Customization Guide](https://github.com/samvera/bulkrax/wiki/Customizing-Bulkrax).
|
157
159
|
|
158
160
|
## How it Works
|
159
161
|
Once you have Bulkrax installed, you will have access to an easy to use interface with which you are able to create, edit, delete, run, and re-run imports and exports.
|
@@ -180,16 +182,21 @@ To delete an importer or exporter, select the delete (x) icon.
|
|
180
182
|
### Downloading an export
|
181
183
|
Once your the exporter has run, a download icon will appear on the exporters menu page.
|
182
184
|
|
185
|
+
## Compatibility
|
186
|
+
|
187
|
+
* Ruby 2.7 or newer is required
|
188
|
+
* Hyrax 2.3 or newer is required
|
189
|
+
|
183
190
|
## Contributing
|
184
191
|
If you're working on a PR for this project, create a feature branch off of `main`.
|
185
192
|
|
186
193
|
This repository follows the [Samvera Community Code of Conduct](https://samvera.atlassian.net/wiki/spaces/samvera/pages/405212316/Code+of+Conduct) and [language recommendations](https://github.com/samvera/maintenance/blob/master/templates/CONTRIBUTING.md#language). Please ***do not*** create a branch called `master` for this repository or as part of your pull request; the branch will either need to be removed or renamed before it can be considered for inclusion in the code base and history of this repository.
|
187
194
|
|
188
195
|
See
|
189
|
-
[CONTRIBUTING.md](https://github.com/samvera
|
196
|
+
[CONTRIBUTING.md](https://github.com/samvera/bulkrax/blob/main/CONTRIBUTING.md)
|
190
197
|
for contributing guidelines.
|
191
198
|
|
192
|
-
We encourage everyone to help improve this project. Bug reports and pull requests are welcome on GitHub at https://github.com/samvera
|
199
|
+
We encourage everyone to help improve this project. Bug reports and pull requests are welcome on GitHub at https://github.com/samvera/bulkrax.
|
193
200
|
|
194
201
|
This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](https://contributor-covenant.org) code of conduct.
|
195
202
|
|
@@ -14,7 +14,7 @@ function hideUnhide(field) {
|
|
14
14
|
}
|
15
15
|
};
|
16
16
|
|
17
|
-
function addRequired(selectedSource) {
|
17
|
+
function addRequired(selectedSource) {
|
18
18
|
selectedSource.addClass('required').attr('required', 'required');
|
19
19
|
selectedSource.parent().addClass('required');
|
20
20
|
}
|
@@ -26,14 +26,14 @@ function removeRequired(allSources) {
|
|
26
26
|
|
27
27
|
// hide all export_source
|
28
28
|
function hide(allSources) {
|
29
|
-
allSources.addClass('
|
30
|
-
allSources.find('#exporter_export_source').addClass('
|
29
|
+
allSources.addClass('d-none');
|
30
|
+
allSources.find('#exporter_export_source').addClass('.d-none').attr('type', 'd-none');
|
31
31
|
}
|
32
32
|
|
33
33
|
// unhide selected export_source
|
34
34
|
function unhideSelected(selectedSource) {
|
35
|
-
selectedSource.removeClass('
|
36
|
-
selectedSource.parent().removeClass('
|
35
|
+
selectedSource.removeClass('d-none').removeAttr('type');
|
36
|
+
selectedSource.parent().removeClass('d-none').removeAttr('type');
|
37
37
|
};
|
38
38
|
|
39
39
|
// add the autocomplete javascript
|
@@ -1,9 +1,26 @@
|
|
1
1
|
<% unless defined?(::Hyku) %>
|
2
2
|
// enables the tabs in the importers/exporters pages.
|
3
3
|
$(document).ready(function() {
|
4
|
-
$('.nav-tabs a').click(function
|
4
|
+
$('.bulkrax-nav-tab-top-margin.nav-tabs a').click(function(e) {
|
5
5
|
e.preventDefault();
|
6
|
-
|
6
|
+
|
7
|
+
// Remove active class from all tabs and hide all tab content
|
8
|
+
$('.bulkrax-nav-tab-top-margin.nav-tabs a').parent().removeClass('active');
|
9
|
+
$('.tab-content .tab-pane').removeClass('active');
|
10
|
+
|
11
|
+
// Add active class to clicked tab and show its content
|
12
|
+
$(this).parent().addClass('active');
|
13
|
+
$($(this).attr('href')).addClass('active');
|
7
14
|
});
|
15
|
+
|
16
|
+
$('#full-errors-tab, #full-errors-tab a').click(function(e) {
|
17
|
+
$('#raw-errors-tab, #bulkrax-raw-toggle-1').removeClass('active');
|
18
|
+
$('#full-errors-tab, #bulkrax-full-toggle-1').addClass('active');
|
19
|
+
})
|
20
|
+
|
21
|
+
$('#raw-errors-tab, #raw-errors-tab a').click(function(e) {
|
22
|
+
$('#full-errors-tab, #bulkrax-full-toggle-1').removeClass('active');
|
23
|
+
$('#raw-errors-tab, #bulkrax-raw-toggle-1').addClass('active');
|
24
|
+
})
|
8
25
|
});
|
9
26
|
<% end %>
|
@@ -123,7 +123,7 @@ module Bulkrax
|
|
123
123
|
def create
|
124
124
|
attrs = transform_attributes
|
125
125
|
@object = klass.new
|
126
|
-
object.reindex_extent = Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX if object.respond_to?(:reindex_extent)
|
126
|
+
object.reindex_extent = Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX if defined?(Hyrax::Adapters::NestingIndexAdapter) && object.respond_to?(:reindex_extent)
|
127
127
|
run_callbacks :save do
|
128
128
|
run_callbacks :create do
|
129
129
|
if klass == Collection
|
@@ -253,7 +253,7 @@ module Bulkrax
|
|
253
253
|
def collection_type(attrs)
|
254
254
|
return attrs if attrs['collection_type_gid'].present?
|
255
255
|
|
256
|
-
attrs['collection_type_gid'] = Hyrax::CollectionType.find_or_create_default_collection_type.
|
256
|
+
attrs['collection_type_gid'] = Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s
|
257
257
|
attrs
|
258
258
|
end
|
259
259
|
|
@@ -40,7 +40,7 @@ module Bulkrax
|
|
40
40
|
|
41
41
|
include DynamicRecordLookup
|
42
42
|
|
43
|
-
queue_as
|
43
|
+
queue_as :import
|
44
44
|
|
45
45
|
# @param parent_identifier [String] Work/Collection ID or Bulkrax::Entry source_identifiers
|
46
46
|
# @param importer_run [Bulkrax::ImporterRun] current importer run (needed to properly update counters)
|
@@ -98,7 +98,7 @@ module Bulkrax
|
|
98
98
|
|
99
99
|
if errors.present?
|
100
100
|
# rubocop:disable Rails/SkipsModelValidations
|
101
|
-
|
101
|
+
ImporterRun.update_counters(importer_run_id, failed_relationships: number_of_failures)
|
102
102
|
# rubocop:enable Rails/SkipsModelValidations
|
103
103
|
|
104
104
|
parent_entry&.set_status_info(errors.last, importer_run)
|
@@ -108,7 +108,7 @@ module Bulkrax
|
|
108
108
|
return false # stop current job from continuing to run after rescheduling
|
109
109
|
else
|
110
110
|
# rubocop:disable Rails/SkipsModelValidations
|
111
|
-
|
111
|
+
ImporterRun.update_counters(importer_run_id, processed_relationships: number_of_successes)
|
112
112
|
# rubocop:enable Rails/SkipsModelValidations
|
113
113
|
end
|
114
114
|
end
|
@@ -158,7 +158,8 @@ module Bulkrax
|
|
158
158
|
end
|
159
159
|
|
160
160
|
def add_to_collection(child_record, parent_record)
|
161
|
-
parent_record.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX)
|
161
|
+
parent_record.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
162
|
+
defined?(Hyrax::Adapters::NestingIndexAdapter)
|
162
163
|
child_record.member_of_collections << parent_record
|
163
164
|
child_record.save!
|
164
165
|
end
|
@@ -2,19 +2,19 @@
|
|
2
2
|
|
3
3
|
module Bulkrax
|
4
4
|
class DeleteJob < ApplicationJob
|
5
|
-
queue_as
|
5
|
+
queue_as :import
|
6
6
|
|
7
|
-
# rubocop:disable Rails/SkipsModelValidations
|
8
7
|
def perform(entry, importer_run)
|
9
8
|
obj = entry.factory.find
|
10
9
|
obj&.delete
|
11
|
-
|
12
|
-
ImporterRun.
|
10
|
+
# rubocop:disable Rails/SkipsModelValidations
|
11
|
+
ImporterRun.increment_counter(:deleted_records, importer_run.id)
|
12
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run.id)
|
13
|
+
# rubocop:enable Rails/SkipsModelValidations
|
13
14
|
entry.save!
|
14
15
|
entry.importer.current_run = ImporterRun.find(importer_run.id)
|
15
16
|
entry.importer.record_status
|
16
17
|
entry.set_status_info("Deleted", ImporterRun.find(importer_run.id))
|
17
18
|
end
|
18
|
-
# rubocop:enable Rails/SkipsModelValidations
|
19
19
|
end
|
20
20
|
end
|
@@ -12,17 +12,17 @@ module Bulkrax
|
|
12
12
|
entry.save
|
13
13
|
rescue StandardError
|
14
14
|
# rubocop:disable Rails/SkipsModelValidations
|
15
|
-
|
16
|
-
|
15
|
+
ExporterRun.increment_counter(:failed_records, args[1])
|
16
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
17
17
|
raise
|
18
18
|
else
|
19
19
|
if entry.failed?
|
20
|
-
|
21
|
-
|
20
|
+
ExporterRun.increment_counter(:failed_records, args[1])
|
21
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
22
22
|
raise entry.reload.current_status.error_class.constantize
|
23
23
|
else
|
24
|
-
|
25
|
-
|
24
|
+
ExporterRun.increment_counter(:processed_records, args[1])
|
25
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
26
26
|
end
|
27
27
|
# rubocop:enable Rails/SkipsModelValidations
|
28
28
|
end
|
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
module Bulkrax
|
4
4
|
class ImportCollectionJob < ApplicationJob
|
5
|
-
queue_as
|
5
|
+
queue_as :import
|
6
6
|
|
7
7
|
# rubocop:disable Rails/SkipsModelValidations
|
8
8
|
def perform(*args)
|
@@ -10,13 +10,13 @@ module Bulkrax
|
|
10
10
|
begin
|
11
11
|
entry.build
|
12
12
|
entry.save!
|
13
|
-
ImporterRun.
|
14
|
-
ImporterRun.
|
15
|
-
ImporterRun.
|
13
|
+
ImporterRun.increment_counter(:processed_records, args[1])
|
14
|
+
ImporterRun.increment_counter(:processed_collections, args[1])
|
15
|
+
ImporterRun.decrement_counter(:enqueued_records, args[1]) unless ImporterRun.find(args[1]).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
16
16
|
rescue => e
|
17
|
-
ImporterRun.
|
18
|
-
ImporterRun.
|
19
|
-
ImporterRun.
|
17
|
+
ImporterRun.increment_counter(:failed_records, args[1])
|
18
|
+
ImporterRun.increment_counter(:failed_collections, args[1])
|
19
|
+
ImporterRun.decrement_counter(:enqueued_records, args[1]) unless ImporterRun.find(args[1]).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
20
20
|
raise e
|
21
21
|
end
|
22
22
|
entry.importer.current_run = ImporterRun.find(args[1])
|
@@ -6,7 +6,7 @@ module Bulkrax
|
|
6
6
|
class ImportFileSetJob < ApplicationJob
|
7
7
|
include DynamicRecordLookup
|
8
8
|
|
9
|
-
queue_as
|
9
|
+
queue_as :import
|
10
10
|
|
11
11
|
attr_reader :importer_run_id
|
12
12
|
|
@@ -21,14 +21,14 @@ module Bulkrax
|
|
21
21
|
entry.build
|
22
22
|
if entry.succeeded?
|
23
23
|
# rubocop:disable Rails/SkipsModelValidations
|
24
|
-
ImporterRun.
|
25
|
-
ImporterRun.
|
24
|
+
ImporterRun.increment_counter(:processed_records, importer_run_id)
|
25
|
+
ImporterRun.increment_counter(:processed_file_sets, importer_run_id)
|
26
26
|
else
|
27
|
-
ImporterRun.
|
28
|
-
ImporterRun.
|
27
|
+
ImporterRun.increment_counter(:failed_records, importer_run_id)
|
28
|
+
ImporterRun.increment_counter(:failed_file_sets, importer_run_id)
|
29
29
|
# rubocop:enable Rails/SkipsModelValidations
|
30
30
|
end
|
31
|
-
ImporterRun.
|
31
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run_id) unless ImporterRun.find(importer_run_id).enqueued_records <= 0 # rubocop:disable Rails/SkipsModelValidations
|
32
32
|
entry.save!
|
33
33
|
entry.importer.current_run = ImporterRun.find(importer_run_id)
|
34
34
|
entry.importer.record_status
|
@@ -40,7 +40,7 @@ module Bulkrax
|
|
40
40
|
if entry.import_attempts < 5
|
41
41
|
ImportFileSetJob.set(wait: (entry.import_attempts + 1).minutes).perform_later(entry_id, importer_run_id)
|
42
42
|
else
|
43
|
-
ImporterRun.
|
43
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run_id) # rubocop:disable Rails/SkipsModelValidations
|
44
44
|
entry.set_status_info(e)
|
45
45
|
end
|
46
46
|
end
|
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
module Bulkrax
|
4
4
|
class ImportWorkJob < ApplicationJob
|
5
|
-
queue_as
|
5
|
+
queue_as :import
|
6
6
|
|
7
7
|
# rubocop:disable Rails/SkipsModelValidations
|
8
8
|
#
|
@@ -23,16 +23,16 @@ module Bulkrax
|
|
23
23
|
entry = Entry.find(entry_id)
|
24
24
|
entry.build
|
25
25
|
if entry.status == "Complete"
|
26
|
-
ImporterRun.
|
27
|
-
ImporterRun.
|
26
|
+
ImporterRun.increment_counter(:processed_records, run_id)
|
27
|
+
ImporterRun.increment_counter(:processed_works, run_id)
|
28
28
|
else
|
29
29
|
# do not retry here because whatever parse error kept you from creating a work will likely
|
30
30
|
# keep preventing you from doing so.
|
31
|
-
ImporterRun.
|
32
|
-
ImporterRun.
|
31
|
+
ImporterRun.increment_counter(:failed_records, run_id)
|
32
|
+
ImporterRun.increment_counter(:failed_works, run_id)
|
33
33
|
end
|
34
34
|
# Regardless of completion or not, we want to decrement the enqueued records.
|
35
|
-
ImporterRun.
|
35
|
+
ImporterRun.decrement_counter(:enqueued_records, run_id) unless ImporterRun.find(run_id).enqueued_records <= 0
|
36
36
|
|
37
37
|
entry.save!
|
38
38
|
entry.importer.current_run = ImporterRun.find(run_id)
|
@@ -15,7 +15,7 @@ module Bulkrax
|
|
15
15
|
def add_collection_type_gid
|
16
16
|
return if self.parsed_metadata['collection_type_gid'].present?
|
17
17
|
|
18
|
-
self.parsed_metadata['collection_type_gid'] = ::Hyrax::CollectionType.find_or_create_default_collection_type.
|
18
|
+
self.parsed_metadata['collection_type_gid'] = ::Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s
|
19
19
|
end
|
20
20
|
end
|
21
21
|
end
|
@@ -88,8 +88,10 @@ module Bulkrax
|
|
88
88
|
|
89
89
|
def validate_record
|
90
90
|
raise StandardError, 'Record not found' if record.nil?
|
91
|
-
|
92
|
-
|
91
|
+
unless importerexporter.parser.required_elements?(record)
|
92
|
+
raise StandardError, "Missing required elements, missing element(s) are: "\
|
93
|
+
"#{importerexporter.parser.missing_elements(record).join(', ')}"
|
94
|
+
end
|
93
95
|
end
|
94
96
|
|
95
97
|
def add_identifier
|
@@ -270,8 +270,8 @@ module Bulkrax
|
|
270
270
|
current_run.invalid_records ||= ""
|
271
271
|
current_run.invalid_records += message
|
272
272
|
current_run.save
|
273
|
-
ImporterRun.
|
274
|
-
ImporterRun.
|
273
|
+
ImporterRun.increment_counter(:failed_records, current_run.id)
|
274
|
+
ImporterRun.decrement_counter(:enqueued_records, current_run.id) unless ImporterRun.find(current_run.id).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
275
275
|
end
|
276
276
|
# rubocop:enable Rails/SkipsModelValidations
|
277
277
|
|
@@ -33,9 +33,10 @@ module Bulkrax
|
|
33
33
|
model_field_mappings.map(&:to_sym).each do |model_mapping|
|
34
34
|
next unless r.key?(model_mapping)
|
35
35
|
|
36
|
-
|
36
|
+
model = r[model_mapping].nil? ? "" : r[model_mapping].strip
|
37
|
+
if model.casecmp('collection').zero?
|
37
38
|
@collections << r
|
38
|
-
elsif
|
39
|
+
elsif model.casecmp('fileset').zero?
|
39
40
|
@file_sets << r
|
40
41
|
else
|
41
42
|
@works << r
|
@@ -67,7 +67,7 @@ module Bulkrax
|
|
67
67
|
metadata = {
|
68
68
|
visibility: 'open'
|
69
69
|
}
|
70
|
-
metadata[:collection_type_gid] = Hyrax::CollectionType.find_or_create_default_collection_type.
|
70
|
+
metadata[:collection_type_gid] = Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s if defined?(::Hyrax)
|
71
71
|
|
72
72
|
collections.each_with_index do |set, index|
|
73
73
|
next unless collection_name == 'all' || collection_name == set.spec
|
@@ -63,7 +63,8 @@ module Bulkrax
|
|
63
63
|
remove_relationships_from_work(obj)
|
64
64
|
end
|
65
65
|
|
66
|
-
obj.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
66
|
+
obj.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
67
|
+
defined?(Hyrax::Adapters::NestingIndexAdapter)
|
67
68
|
obj.save!
|
68
69
|
end
|
69
70
|
end
|
@@ -33,8 +33,8 @@
|
|
33
33
|
label: t('bulkrax.exporter.labels.importer'),
|
34
34
|
required: true,
|
35
35
|
prompt: 'Select from the list',
|
36
|
-
label_html: { class: 'importer export-source-option
|
37
|
-
input_html: { class: 'importer export-source-option
|
36
|
+
label_html: { class: 'importer export-source-option d-none' },
|
37
|
+
input_html: { class: 'importer export-source-option d-none form-control' },
|
38
38
|
collection: form.object.importers_list.sort %>
|
39
39
|
|
40
40
|
<%= form.input :export_source_collection,
|
@@ -42,9 +42,9 @@
|
|
42
42
|
label: t('bulkrax.exporter.labels.collection'),
|
43
43
|
required: true,
|
44
44
|
placeholder: @collection&.title&.first,
|
45
|
-
label_html: { class: 'collection export-source-option
|
45
|
+
label_html: { class: 'collection export-source-option d-none' },
|
46
46
|
input_html: {
|
47
|
-
class: 'collection export-source-option
|
47
|
+
class: 'collection export-source-option d-none form-control',
|
48
48
|
data: {
|
49
49
|
'autocomplete-url' => '/authorities/search/collections',
|
50
50
|
'autocomplete' => 'collection'
|
@@ -56,8 +56,8 @@
|
|
56
56
|
label: t('bulkrax.exporter.labels.worktype'),
|
57
57
|
required: true,
|
58
58
|
prompt: 'Select from the list',
|
59
|
-
label_html: { class: 'worktype export-source-option
|
60
|
-
input_html: { class: 'worktype export-source-option
|
59
|
+
label_html: { class: 'worktype export-source-option d-none' },
|
60
|
+
input_html: { class: 'worktype export-source-option d-none form-control' },
|
61
61
|
collection: Bulkrax.curation_concerns.map { |cc| [cc.to_s, cc.to_s] } %>
|
62
62
|
|
63
63
|
<%= form.input :limit,
|
@@ -80,7 +80,7 @@
|
|
80
80
|
as: :boolean,
|
81
81
|
label: t('bulkrax.exporter.labels.filter_by_date') %>
|
82
82
|
|
83
|
-
<div id="date_filter_picker" class="
|
83
|
+
<div id="date_filter_picker" class="d-none">
|
84
84
|
<%= form.input :start_date,
|
85
85
|
as: :date,
|
86
86
|
label: t('bulkrax.exporter.labels.start_date'),
|
@@ -136,13 +136,13 @@
|
|
136
136
|
// get the date filter option and show the corresponding date selectors
|
137
137
|
$('.exporter_date_filter').change(function () {
|
138
138
|
if ($('.exporter_date_filter').find(".boolean").is(":checked"))
|
139
|
-
$('#date_filter_picker').removeClass('
|
139
|
+
$('#date_filter_picker').removeClass('d-none');
|
140
140
|
else
|
141
|
-
$('#date_filter_picker').addClass('
|
141
|
+
$('#date_filter_picker').addClass('d-none');
|
142
142
|
});
|
143
143
|
|
144
144
|
if ($('.exporter_date_filter').find(".boolean").is(":checked"))
|
145
|
-
$('#date_filter_picker').removeClass('
|
145
|
+
$('#date_filter_picker').removeClass('d-none');
|
146
146
|
});
|
147
147
|
});
|
148
148
|
</script>
|
@@ -96,9 +96,9 @@
|
|
96
96
|
<div class="bulkrax-nav-tab-bottom-margin">
|
97
97
|
<!-- Nav tabs -->
|
98
98
|
<ul class="bulkrax-nav-tab-top-margin tab-nav nav nav-tabs" role="tablist">
|
99
|
-
<li role="presentation" class=
|
100
|
-
<li role="presentation"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.collection_entries') %></a></li>
|
101
|
-
<li role="presentation"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.file_set_entries') %></a></li>
|
99
|
+
<li role="presentation" class="nav-link active"><a href="#work-entries" aria-controls="work-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.work_entries') %></a></li>
|
100
|
+
<li role="presentation" class="nav-link"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.collection_entries') %></a></li>
|
101
|
+
<li role="presentation" class="nav-link"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.file_set_entries') %></a></li>
|
102
102
|
</ul>
|
103
103
|
<!-- Tab panes -->
|
104
104
|
<div class="tab-content outline">
|
@@ -76,9 +76,9 @@
|
|
76
76
|
<div class="bulkrax-nav-tab-bottom-margin">
|
77
77
|
<!-- Nav tabs -->
|
78
78
|
<ul class="bulkrax-nav-tab-top-margin tab-nav nav nav-tabs" role="tablist">
|
79
|
-
<li role="presentation" class=
|
80
|
-
<li role="presentation"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.collection_entries') %></a></li>
|
81
|
-
<li role="presentation"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.file_set_entries') %></a></li>
|
79
|
+
<li role="presentation" class="nav-link active"><a href="#work-entries" aria-controls="work-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.work_entries') %></a></li>
|
80
|
+
<li role="presentation" class="nav-link"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.collection_entries') %></a></li>
|
81
|
+
<li role="presentation" class="nav-link"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.file_set_entries') %></a></li>
|
82
82
|
</ul>
|
83
83
|
<!-- Tab panes -->
|
84
84
|
<div class="tab-content outline">
|
@@ -13,8 +13,8 @@
|
|
13
13
|
<div class="bulkrax-nav-tab-bottom-margin">
|
14
14
|
<!-- Toggle buttons -->
|
15
15
|
<div class="btn-group pull-right" role="group" aria-label="...">
|
16
|
-
<button type="button" class="btn btn-default active"><a href="#bulkrax-full-toggle-1" aria-controls="bulkrax-full-toggle-1" role="tab" data-toggle="tab">Full</a></button>
|
17
|
-
<button type="button" class="btn btn-default"><a href="#bulkrax-raw-toggle-1" aria-controls="bulkrax-raw-toggle-1" role="tab" data-toggle="tab">Raw</a></button>
|
16
|
+
<button id="full-errors-tab" type="button" class="btn btn-default active"><a href="#bulkrax-full-toggle-1" aria-controls="bulkrax-full-toggle-1" role="tab" data-toggle="tab">Full</a></button>
|
17
|
+
<button id="raw-errors-tab" type="button" class="btn btn-default"><a href="#bulkrax-raw-toggle-1" aria-controls="bulkrax-raw-toggle-1" role="tab" data-toggle="tab">Raw</a></button>
|
18
18
|
</div>
|
19
19
|
<!-- Tab panes -->
|
20
20
|
<div class="tab-content">
|
@@ -1,12 +1,20 @@
|
|
1
1
|
<% if current_ability.can_import_works? %>
|
2
|
-
<%= menu.nav_link(
|
3
|
-
|
2
|
+
<%= menu.nav_link(
|
3
|
+
bulkrax.importers_path,
|
4
|
+
title: t('bulkrax.admin.sidebar.importers'),
|
5
|
+
class: "nav-link",
|
6
|
+
onclick: "dontChangeAccordion(event);"
|
7
|
+
) do %>
|
4
8
|
<span class="fa fa-cloud-upload" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.importers') %></span>
|
5
9
|
<% end %>
|
6
10
|
<% end %>
|
7
11
|
<% if current_ability.can_export_works? %>
|
8
|
-
<%= menu.nav_link(
|
9
|
-
|
12
|
+
<%= menu.nav_link(
|
13
|
+
bulkrax.exporters_path,
|
14
|
+
title: t('bulkrax.admin.sidebar.exporters'),
|
15
|
+
class: "nav-link",
|
16
|
+
onclick: "dontChangeAccordion(event);"
|
17
|
+
) do %>
|
10
18
|
<span class="fa fa-cloud-download" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.exporters') %></span>
|
11
19
|
<% end %>
|
12
20
|
<% end %>
|
@@ -6,11 +6,13 @@ end
|
|
6
6
|
|
7
7
|
class ChangeImporterAndExporterToPolymorphic < ActiveRecord::Migration[5.1]
|
8
8
|
def change
|
9
|
-
|
10
|
-
remove_foreign_key :bulkrax_entries, column: :importer_id
|
11
|
-
|
12
|
-
|
9
|
+
begin # deal with odd bug around foreign keys in ci
|
10
|
+
remove_foreign_key :bulkrax_entries, column: :importer_id if foreign_key_exists?(:bulkrax_entries, column: :importer_id)
|
11
|
+
rescue ArgumentError
|
12
|
+
# do nothing
|
13
13
|
end
|
14
|
+
remove_index :bulkrax_entries, :importer_id if index_exists?(:bulkrax_entries, :importer_id)
|
15
|
+
rename_column :bulkrax_entries, :importer_id, :importerexporter_id if column_exists?(:bulkrax_entries, :importer_id)
|
14
16
|
add_column :bulkrax_entries, :importerexporter_type, :string, after: :id, default: 'Bulkrax::Importer' unless column_exists?(:bulkrax_entries, :importerexporter_type)
|
15
17
|
end
|
16
18
|
end
|
@@ -10,13 +10,17 @@ module Bulkrax
|
|
10
10
|
#
|
11
11
|
# This module came about through a desire to expose a quick means of vetting the accuracy of the
|
12
12
|
# different parsers.
|
13
|
+
#
|
14
|
+
# @see .entry_for
|
13
15
|
module EntrySpecHelper
|
14
16
|
##
|
15
17
|
# @api public
|
16
18
|
# @since v5.0.1
|
17
19
|
#
|
18
|
-
# The purpose of this method is encapsulate the logic of creating the appropriate
|
19
|
-
# object based on the given data, identifier, and parser_class_name.
|
20
|
+
# The purpose of this method is encapsulate the logic of creating the appropriate
|
21
|
+
# {Bulkrax::Entry} object based on the given data, identifier, and parser_class_name. Due to
|
22
|
+
# the different means of instantiation of {Bulkrax::Entry} subclasses, there are several
|
23
|
+
# optional parameters.
|
20
24
|
#
|
21
25
|
# From that entry, you should be able to test how {Bulkrax::Entry#build_metadata} populates the
|
22
26
|
# {Bulkrax::Entry#parsed_metadata} variable. Other uses may emerge.
|
@@ -29,9 +33,47 @@ module Bulkrax
|
|
29
33
|
# @param parser_class_name [String] The name of the parser class you're wanting to test.
|
30
34
|
# @param type [Sybmol] The type of entry (e.g. :entry, :collection, :file_set) for testing.
|
31
35
|
# @param options [Hash<Symbol,Object>] these are to be passed along into the instantiation of
|
32
|
-
# the various classes.
|
36
|
+
# the various classes.
|
37
|
+
# @option options [String] importer_name (Optional) The name of the test importer. One will be
|
38
|
+
# auto-assigned if unprovided.
|
39
|
+
# @option options [String] importer_admin_set_id (Optional) The ID of an admin set to deposit
|
40
|
+
# into. One will be auto-assigned if unprovided. And this admin set does not need to
|
41
|
+
# be persisted nor exist. It is simply a required parameter for instantiating an
|
42
|
+
# importer.
|
43
|
+
# @option options [User] user (Optional) The user who is performing the import. One will be
|
44
|
+
# auto-assigned if unprovided. The user does not need to be persisted. It is simply a
|
45
|
+
# required parameter for instantiating an importer
|
46
|
+
# @option options [Integer] limit (Optional) You really shouldn't need to set this, but for
|
47
|
+
# completeness it is provided.
|
48
|
+
# @option options [Hash<String, Object>] importer_field_mappings Each parser class may require
|
49
|
+
# different field mappings. See the given examples for more details.
|
50
|
+
#
|
51
|
+
# @return [Bulkrax::Entry] a subclass of {Bulkrax::Entry} based on the application's
|
52
|
+
# configuration. It would behoove you to write a spec regarding the returned entry's
|
53
|
+
# class.
|
54
|
+
#
|
55
|
+
# @example
|
56
|
+
# entry = Bulkrax::EntrySpecHelper.entry_for(
|
57
|
+
# data: { source_identifier: "123", title: "Hello World" },
|
58
|
+
# parser_class_name: "Bulkrax::CsvParser",
|
59
|
+
# importer_field_mappings: { 'import_file_path' => "path/to/file.csv" }
|
60
|
+
# )
|
61
|
+
#
|
62
|
+
# @note In the case of the Bulkrax::CsvParser, the :data keyword is a Hash, where the keys are
|
63
|
+
# the column name of the CSV you're importing. The 'import_file_path' is a path to a CSV
|
64
|
+
# file. That CSV's columns does not need to match the :data's keys, though there may be
|
65
|
+
# required headers on that CSV based on the parser implementation.
|
33
66
|
#
|
34
|
-
# @
|
67
|
+
# @example
|
68
|
+
# entry = Bulkrax::EntrySpecHelper.entry_for(
|
69
|
+
# identifier: identifier,
|
70
|
+
# data: File.read("/path/to/some/file.xml"),
|
71
|
+
# parser_class_name: "Bulkrax::OaiDcParser",
|
72
|
+
# parser_fields: { "base_url" => "http://oai.adventistdigitallibrary.org/OAI-script" }
|
73
|
+
# )
|
74
|
+
#
|
75
|
+
# @note In the case of an OaiParser, the :data keyword should be a String. And you'll need to
|
76
|
+
# provide a :parser_fields with a "base_url".
|
35
77
|
def self.entry_for(data:, identifier:, parser_class_name:, type: :entry, **options)
|
36
78
|
importer = importer_for(parser_class_name: parser_class_name, **options)
|
37
79
|
entry_type_method_name = ENTRY_TYPE_TO_METHOD_NAME_MAP.fetch(type)
|
@@ -58,6 +100,7 @@ module Bulkrax
|
|
58
100
|
**options)
|
59
101
|
end
|
60
102
|
|
103
|
+
##
|
61
104
|
# @api public
|
62
105
|
#
|
63
106
|
# @param parser_class_name [String]
|
@@ -121,16 +164,6 @@ module Bulkrax
|
|
121
164
|
end
|
122
165
|
private_class_method :importer_for
|
123
166
|
|
124
|
-
##
|
125
|
-
# @api private
|
126
|
-
#
|
127
|
-
# @param data [Hash<Symbol,String>] we're expecting a hash with keys that are symbols and then
|
128
|
-
# values that are strings.
|
129
|
-
#
|
130
|
-
# @return [Bulkrax::CsvEntry]
|
131
|
-
#
|
132
|
-
# @note As a foible of this implementation, you'll need to include along a CSV to establish the
|
133
|
-
# columns that you'll parse (e.g. the first row
|
134
167
|
def self.build_csv_entry_for(importer:, data:, identifier:, entry_class:, **_options)
|
135
168
|
entry_class.new(
|
136
169
|
importerexporter: importer,
|
@@ -138,13 +171,8 @@ module Bulkrax
|
|
138
171
|
raw_metadata: data
|
139
172
|
)
|
140
173
|
end
|
174
|
+
private_class_method :build_csv_entry_for
|
141
175
|
|
142
|
-
##
|
143
|
-
# @api private
|
144
|
-
#
|
145
|
-
# @param data [String] we're expecting a string that is well-formed XML for OAI parsing.
|
146
|
-
#
|
147
|
-
# @return [Bulkrax::OaiEntry]
|
148
176
|
def self.build_oai_entry_for(importer:, data:, identifier:, entry_class:, **options)
|
149
177
|
# The raw record assumes we take the XML data, parse it and then send that to the
|
150
178
|
# OAI::GetRecordResponse object.
|
@@ -165,13 +193,8 @@ module Bulkrax
|
|
165
193
|
raw_metadata: raw_metadata
|
166
194
|
)
|
167
195
|
end
|
196
|
+
private_class_method :build_oai_entry_for
|
168
197
|
|
169
|
-
##
|
170
|
-
# @api private
|
171
|
-
#
|
172
|
-
# @param data [String] we're expecting a string that is well-formed XML.
|
173
|
-
#
|
174
|
-
# @return [Bulkrax::XmlEntry]
|
175
198
|
def self.build_xml_entry_for(importer:, data:, identifier:, entry_class:, **options)
|
176
199
|
raw_metadata = {
|
177
200
|
importer.parser.source_identifier.to_s => identifier,
|
@@ -186,5 +209,6 @@ module Bulkrax
|
|
186
209
|
raw_metadata: raw_metadata
|
187
210
|
)
|
188
211
|
end
|
212
|
+
private_class_method :build_xml_entry_for
|
189
213
|
end
|
190
214
|
end
|
data/lib/bulkrax/version.rb
CHANGED
data/lib/bulkrax.rb
CHANGED
@@ -1,16 +1,16 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
|
3
|
+
require "bulkrax/version"
|
4
|
+
require "bulkrax/engine"
|
5
|
+
require 'active_support/all'
|
6
6
|
|
7
|
-
|
7
|
+
# rubocop:disable Metrics/ModuleLength
|
8
8
|
module Bulkrax
|
9
|
-
|
10
|
-
|
9
|
+
extend self # rubocop:disable Style/ModuleFunction
|
10
|
+
extend Forwardable
|
11
11
|
|
12
|
-
|
13
|
-
|
12
|
+
##
|
13
|
+
# @api public
|
14
14
|
class Configuration
|
15
15
|
attr_accessor :api_definition,
|
16
16
|
:curation_concerns,
|
@@ -35,15 +35,6 @@ module Bulkrax
|
|
35
35
|
:reserved_properties,
|
36
36
|
:server_name
|
37
37
|
|
38
|
-
attr_writer :ingest_queue_name
|
39
|
-
##
|
40
|
-
# @return [String, Proc]
|
41
|
-
def ingest_queue_name
|
42
|
-
return @ingest_queue_name if @ingest_queue_name.present?
|
43
|
-
return Hyrax.config.ingest_queue_name if defined?(Hyrax)
|
44
|
-
:import
|
45
|
-
end
|
46
|
-
|
47
38
|
attr_writer :use_locking
|
48
39
|
|
49
40
|
def use_locking
|
@@ -54,61 +45,61 @@ module Bulkrax
|
|
54
45
|
alias use_locking? use_locking
|
55
46
|
end
|
56
47
|
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
48
|
+
def config
|
49
|
+
@config ||= Configuration.new
|
50
|
+
yield @config if block_given?
|
51
|
+
@config
|
52
|
+
end
|
53
|
+
alias setup config
|
63
54
|
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
55
|
+
def_delegators :@config,
|
56
|
+
:api_definition,
|
57
|
+
:api_definition=,
|
58
|
+
:curation_concerns,
|
59
|
+
:curation_concerns=,
|
60
|
+
:default_field_mapping,
|
61
|
+
:default_field_mapping=,
|
62
|
+
:default_work_type,
|
63
|
+
:default_work_type=,
|
64
|
+
:export_path,
|
65
|
+
:export_path=,
|
66
|
+
:field_mappings,
|
67
|
+
:field_mappings=,
|
68
|
+
:file_model_class,
|
69
|
+
:file_model_class=,
|
70
|
+
:fill_in_blank_source_identifiers,
|
71
|
+
:fill_in_blank_source_identifiers=,
|
72
|
+
:generated_metadata_mapping,
|
73
|
+
:generated_metadata_mapping=,
|
74
|
+
:import_path,
|
75
|
+
:import_path=,
|
76
|
+
:multi_value_element_join_on,
|
77
|
+
:multi_value_element_join_on=,
|
78
|
+
:multi_value_element_split_on,
|
79
|
+
:multi_value_element_split_on=,
|
80
|
+
:object_factory,
|
81
|
+
:object_factory=,
|
82
|
+
:parsers,
|
83
|
+
:parsers=,
|
84
|
+
:qa_controlled_properties,
|
85
|
+
:qa_controlled_properties=,
|
86
|
+
:related_children_field_mapping,
|
87
|
+
:related_children_field_mapping=,
|
88
|
+
:related_parents_field_mapping,
|
89
|
+
:related_parents_field_mapping=,
|
90
|
+
:relationship_job_class,
|
91
|
+
:relationship_job_class=,
|
92
|
+
:removed_image_path,
|
93
|
+
:removed_image_path=,
|
94
|
+
:required_elements,
|
95
|
+
:required_elements=,
|
96
|
+
:reserved_properties,
|
97
|
+
:reserved_properties=,
|
98
|
+
:server_name,
|
99
|
+
:server_name=,
|
100
|
+
:use_locking,
|
101
|
+
:use_locking=,
|
102
|
+
:use_locking?
|
112
103
|
|
113
104
|
config do |conf|
|
114
105
|
conf.parsers = [
|
@@ -147,149 +138,149 @@ module Bulkrax
|
|
147
138
|
# Based on Hyrax CoreMetadata && BasicMetadata
|
148
139
|
# Override at application level to change
|
149
140
|
conf.field_mappings = {
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
168
|
-
|
141
|
+
"Bulkrax::OaiDcParser" => {
|
142
|
+
"contributor" => { from: ["contributor"] },
|
143
|
+
# no appropriate mapping for coverage (based_near needs id)
|
144
|
+
# ""=>{:from=>["coverage"]},
|
145
|
+
"creator" => { from: ["creator"] },
|
146
|
+
"date_created" => { from: ["date"] },
|
147
|
+
"description" => { from: ["description"] },
|
148
|
+
# no appropriate mapping for format
|
149
|
+
# ""=>{:from=>["format"]},
|
150
|
+
"identifier" => { from: ["identifier"] },
|
151
|
+
"language" => { from: ["language"], parsed: true },
|
152
|
+
"publisher" => { from: ["publisher"] },
|
153
|
+
"related_url" => { from: ["relation"] },
|
154
|
+
"rights_statement" => { from: ["rights"] },
|
155
|
+
"source" => { from: ["source"] },
|
156
|
+
"subject" => { from: ["subject"], parsed: true },
|
157
|
+
"title" => { from: ["title"] },
|
158
|
+
"resource_type" => { from: ["type"], parsed: true },
|
159
|
+
"remote_files" => { from: ["thumbnail_url"], parsed: true }
|
169
160
|
},
|
170
|
-
|
171
|
-
|
172
|
-
|
161
|
+
"Bulkrax::OaiQualifiedDcParser" => {
|
162
|
+
"abstract" => { from: ["abstract"] },
|
163
|
+
"alternative_title" => { from: ["alternative"] },
|
173
164
|
"bibliographic_citation" => { from: ["bibliographicCitation"] },
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
|
192
|
-
|
193
|
-
|
194
|
-
|
165
|
+
"contributor" => { from: ["contributor"] },
|
166
|
+
"creator" => { from: ["creator"] },
|
167
|
+
"date_created" => { from: ["created"] },
|
168
|
+
"description" => { from: ["description"] },
|
169
|
+
"language" => { from: ["language"] },
|
170
|
+
"license" => { from: ["license"] },
|
171
|
+
"publisher" => { from: ["publisher"] },
|
172
|
+
"related_url" => { from: ["relation"] },
|
173
|
+
"rights_holder" => { from: ["rightsHolder"] },
|
174
|
+
"rights_statement" => { from: ["rights"] },
|
175
|
+
"source" => { from: ["source"] },
|
176
|
+
"subject" => { from: ["subject"], parsed: true },
|
177
|
+
"title" => { from: ["title"] },
|
178
|
+
"resource_type" => { from: ["type"], parsed: true },
|
179
|
+
"remote_files" => { from: ["thumbnail_url"], parsed: true }
|
180
|
+
},
|
181
|
+
# When empty, a default_field_mapping will be generated
|
182
|
+
"Bulkrax::CsvParser" => {},
|
183
|
+
'Bulkrax::BagitParser' => {},
|
184
|
+
'Bulkrax::XmlParser' => {}
|
185
|
+
}
|
195
186
|
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
|
206
|
-
|
207
|
-
|
208
|
-
|
209
|
-
|
187
|
+
# Lambda to set the default field mapping
|
188
|
+
conf.default_field_mapping = lambda do |field|
|
189
|
+
return if field.blank?
|
190
|
+
{
|
191
|
+
field.to_s =>
|
192
|
+
{
|
193
|
+
from: [field.to_s],
|
194
|
+
split: false,
|
195
|
+
parsed: Bulkrax::ApplicationMatcher.method_defined?("parse_#{field}"),
|
196
|
+
if: nil,
|
197
|
+
excluded: false
|
198
|
+
}
|
199
|
+
}
|
200
|
+
end
|
210
201
|
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
|
224
|
-
|
225
|
-
|
226
|
-
|
227
|
-
|
228
|
-
|
229
|
-
|
230
|
-
|
202
|
+
# Properties that should not be used in imports. They are reserved for use by Hyrax.
|
203
|
+
conf.reserved_properties = %w[
|
204
|
+
create_date
|
205
|
+
modified_date
|
206
|
+
date_modified
|
207
|
+
date_uploaded
|
208
|
+
depositor
|
209
|
+
arkivo_checksum
|
210
|
+
has_model
|
211
|
+
head
|
212
|
+
label
|
213
|
+
import_url
|
214
|
+
on_behalf_of
|
215
|
+
proxy_depositor
|
216
|
+
owner
|
217
|
+
state
|
218
|
+
tail
|
219
|
+
original_url
|
220
|
+
relative_path
|
221
|
+
]
|
231
222
|
|
232
|
-
|
233
|
-
|
234
|
-
|
235
|
-
|
236
|
-
|
223
|
+
# List of Questioning Authority properties that are controlled via YAML files in
|
224
|
+
# the config/authorities/ directory. For example, the :rights_statement property
|
225
|
+
# is controlled by the active terms in config/authorities/rights_statements.yml
|
226
|
+
conf.qa_controlled_properties = %w[rights_statement license]
|
227
|
+
end
|
237
228
|
|
238
|
-
|
239
|
-
|
240
|
-
|
241
|
-
|
242
|
-
|
243
|
-
|
244
|
-
|
245
|
-
|
246
|
-
|
229
|
+
def api_definition
|
230
|
+
@api_definition ||= ActiveSupport::HashWithIndifferentAccess.new(
|
231
|
+
YAML.safe_load(
|
232
|
+
ERB.new(
|
233
|
+
File.read(Rails.root.join('config', 'bulkrax_api.yml'))
|
234
|
+
).result
|
235
|
+
)
|
236
|
+
)
|
237
|
+
end
|
247
238
|
|
248
|
-
|
249
|
-
|
250
|
-
|
251
|
-
|
252
|
-
|
253
|
-
|
254
|
-
|
255
|
-
|
239
|
+
DEFAULT_MULTI_VALUE_ELEMENT_JOIN_ON = ' | '
|
240
|
+
# Specify the delimiter for joining an attribute's multi-value array into a string.
|
241
|
+
#
|
242
|
+
# @note the specific delimiter should likely be present in the multi_value_element_split_on
|
243
|
+
# expression.
|
244
|
+
def multi_value_element_join_on
|
245
|
+
@multi_value_element_join_on ||= DEFAULT_MULTI_VALUE_ELEMENT_JOIN_ON
|
246
|
+
end
|
256
247
|
|
257
|
-
|
258
|
-
|
259
|
-
|
260
|
-
|
261
|
-
|
262
|
-
|
248
|
+
DEFAULT_MULTI_VALUE_ELEMENT_SPLIT_ON = /\s*[:;|]\s*/.freeze
|
249
|
+
# @return [RegexClass] the regular express to use to "split" an attribute's values. If set to
|
250
|
+
# `true` use the DEFAULT_MULTI_VALUE_ELEMENT_JOIN_ON.
|
251
|
+
#
|
252
|
+
# @note The "true" value is to preserve backwards compatibility.
|
253
|
+
# @see DEFAULT_MULTI_VALUE_ELEMENT_JOIN_ON
|
263
254
|
def multi_value_element_split_on
|
264
|
-
|
265
|
-
|
255
|
+
if @multi_value_element_join_on.is_a?(TrueClass)
|
256
|
+
DEFAULT_MULTI_VALUE_ELEMENT_SPLIT_ON
|
266
257
|
else
|
267
|
-
|
258
|
+
@multi_value_element_split_on ||= DEFAULT_MULTI_VALUE_ELEMENT_SPLIT_ON
|
268
259
|
end
|
269
260
|
end
|
270
261
|
|
271
|
-
|
272
|
-
|
273
|
-
|
274
|
-
|
275
|
-
|
276
|
-
|
277
|
-
|
278
|
-
|
279
|
-
|
280
|
-
|
262
|
+
# Responsible for stripping hidden characters from the given string.
|
263
|
+
#
|
264
|
+
# @param value [#to_s]
|
265
|
+
# @return [String] with hidden characters removed
|
266
|
+
#
|
267
|
+
# @see https://github.com/samvera-labs/bulkrax/issues/688
|
268
|
+
def normalize_string(value)
|
269
|
+
# Removing [Byte Order Mark (BOM)](https://en.wikipedia.org/wiki/Byte_order_mark)
|
270
|
+
value.to_s.delete("\xEF\xBB\xBF")
|
271
|
+
end
|
281
272
|
|
282
|
-
|
283
|
-
|
273
|
+
def fallback_user_for_importer_exporter_processing
|
274
|
+
return User.batch_user if defined?(Hyrax) && User.respond_to?(:batch_user)
|
284
275
|
|
285
|
-
|
286
|
-
|
276
|
+
raise "We have no fallback user available for Bulkrax.fallback_user_for_importer_exporter_processing"
|
277
|
+
end
|
287
278
|
|
288
|
-
|
289
|
-
|
290
|
-
|
291
|
-
|
292
|
-
|
279
|
+
# This class confirms to the Active::Support.serialize interface. It's job is to ensure that we
|
280
|
+
# don't have keys with the tricksy Byte Order Mark character.
|
281
|
+
#
|
282
|
+
# @see https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html#method-i-serialize
|
283
|
+
class NormalizedJson
|
293
284
|
def self.normalize_keys(hash)
|
294
285
|
return hash unless hash.respond_to?(:each_pair)
|
295
286
|
returning_value = {}
|
@@ -301,18 +292,18 @@ module Bulkrax
|
|
301
292
|
|
302
293
|
# When we write the serialized data to the database, we "dump" the value into that database
|
303
294
|
# column.
|
304
|
-
|
305
|
-
|
306
|
-
|
295
|
+
def self.dump(value)
|
296
|
+
JSON.dump(normalize_keys(value))
|
297
|
+
end
|
307
298
|
|
308
299
|
# When we load the serialized data from the database, we pass the database's value into "load"
|
309
300
|
# function.
|
310
301
|
#
|
311
302
|
# rubocop:disable Security/JSONLoad
|
312
|
-
|
313
|
-
|
314
|
-
|
303
|
+
def self.load(string)
|
304
|
+
normalize_keys(JSON.load(string))
|
305
|
+
end
|
315
306
|
# rubocop:enable Security/JSONLoad
|
316
|
-
|
307
|
+
end
|
317
308
|
end
|
318
|
-
|
309
|
+
# rubocop:disable Metrics/ModuleLength
|
@@ -80,6 +80,7 @@ Bulkrax.setup do |config|
|
|
80
80
|
end
|
81
81
|
|
82
82
|
# Sidebar for hyrax 3+ support
|
83
|
+
# rubocop:disable Style/IfUnlessModifier
|
83
84
|
if Object.const_defined?(:Hyrax) && ::Hyrax::DashboardController&.respond_to?(:sidebar_partials)
|
84
85
|
Hyrax::DashboardController.sidebar_partials[:repository_content] << "hyrax/dashboard/sidebar/bulkrax_sidebar_additions"
|
85
86
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: bulkrax
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 5.
|
4
|
+
version: 5.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Rob Kaufman
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2023-11-29 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rails
|
@@ -52,20 +52,6 @@ dependencies:
|
|
52
52
|
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
-
- !ruby/object:Gem::Dependency
|
56
|
-
name: dry-monads
|
57
|
-
requirement: !ruby/object:Gem::Requirement
|
58
|
-
requirements:
|
59
|
-
- - "~>"
|
60
|
-
- !ruby/object:Gem::Version
|
61
|
-
version: 1.4.0
|
62
|
-
type: :runtime
|
63
|
-
prerelease: false
|
64
|
-
version_requirements: !ruby/object:Gem::Requirement
|
65
|
-
requirements:
|
66
|
-
- - "~>"
|
67
|
-
- !ruby/object:Gem::Version
|
68
|
-
version: 1.4.0
|
69
55
|
- !ruby/object:Gem::Dependency
|
70
56
|
name: iso8601
|
71
57
|
requirement: !ruby/object:Gem::Requirement
|