bulkrax 5.4.0 → 5.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +13 -6
- data/app/assets/javascripts/bulkrax/exporters.js +5 -5
- data/app/assets/javascripts/bulkrax/navtabs.js.erb +19 -2
- data/app/factories/bulkrax/object_factory.rb +7 -3
- data/app/jobs/bulkrax/create_relationships_job.rb +4 -3
- data/app/jobs/bulkrax/delete_job.rb +4 -4
- data/app/jobs/bulkrax/export_work_job.rb +6 -6
- data/app/jobs/bulkrax/import_collection_job.rb +6 -6
- data/app/jobs/bulkrax/import_file_set_job.rb +6 -6
- data/app/jobs/bulkrax/import_work_job.rb +5 -5
- data/app/models/bulkrax/csv_collection_entry.rb +1 -1
- data/app/models/bulkrax/csv_entry.rb +4 -2
- data/app/parsers/bulkrax/application_parser.rb +2 -2
- data/app/parsers/bulkrax/csv_parser.rb +3 -2
- data/app/parsers/bulkrax/oai_dc_parser.rb +1 -1
- data/app/services/bulkrax/remove_relationships_for_importer.rb +2 -1
- data/app/views/bulkrax/exporters/_form.html.erb +10 -10
- data/app/views/bulkrax/exporters/show.html.erb +3 -3
- data/app/views/bulkrax/importers/_oai_fields.html.erb +1 -1
- data/app/views/bulkrax/importers/show.html.erb +3 -3
- data/app/views/bulkrax/shared/_bulkrax_errors.html.erb +2 -2
- data/app/views/hyrax/dashboard/sidebar/_bulkrax_sidebar_additions.html.erb +12 -4
- data/db/migrate/20190731114016_change_importer_and_exporter_to_polymorphic.rb +6 -4
- data/lib/bulkrax/entry_spec_helper.rb +50 -26
- data/lib/bulkrax/version.rb +1 -1
- data/lib/generators/bulkrax/templates/config/initializers/bulkrax.rb +1 -0
- metadata +2 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5935f574678e9ff331fb1c3e637512e81aaf133acd44ac6ba413828b663c3cf1
|
4
|
+
data.tar.gz: d4a44f4fe47171d22c131bb836ec348091c88fcaa011a2e3a61b4234c8bb3be3
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d14475603489ad27d29db6536e4e6d50e3d3393845724a7d85cf0c674804f3b42fbe1430ff9eea0d7e16d5f0482760ca1615e5a55b66a791ff5eab60f1950ea6
|
7
|
+
data.tar.gz: e593a46744c543b2d9e140010178272b470427f726d3f91504b2e6969a15632f5cf467abe375f588b10b2b9821bcdc3b23730ecccb071b5e0411bbc4e794eb9c
|
data/README.md
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
![Test Suite](https://github.com/samvera/bulkrax/actions/workflows/test.yml/badge.svg)
|
2
|
+
![Test Suite](https://github.com/samvera/bulkrax/actions/workflows/lint.yml/badge.svg)
|
1
3
|
# Bulkrax
|
2
4
|
Bulkrax is a batteries included importer for Samvera applications. It currently includes support for OAI-PMH (DC and Qualified DC) and CSV out of the box. It is also designed to be extensible, allowing you to easily add new importers in to your application or to include them with other gems. Bulkrax provides a full admin interface including creating, editing, scheduling and reviewing imports.
|
3
5
|
|
@@ -10,7 +12,7 @@ Add this line to your application's Gemfile:
|
|
10
12
|
```ruby
|
11
13
|
gem 'bulkrax'
|
12
14
|
# or if using from github
|
13
|
-
gem 'bulkrax', git: 'https://github.com/samvera
|
15
|
+
gem 'bulkrax', git: 'https://github.com/samvera/bulkrax.git', branch: 'main'
|
14
16
|
```
|
15
17
|
|
16
18
|
And then execute:
|
@@ -72,7 +74,7 @@ Bulkrax.setup do |config|
|
|
72
74
|
end
|
73
75
|
```
|
74
76
|
|
75
|
-
The [configuration guide](https://github.com/samvera
|
77
|
+
The [configuration guide](https://github.com/samvera/bulkrax/wiki/Configuring-Bulkrax) provides detailed instructions on the various available configurations.
|
76
78
|
|
77
79
|
Example:
|
78
80
|
|
@@ -122,7 +124,7 @@ It's unlikely that the incoming import data has fields that exactly match those
|
|
122
124
|
|
123
125
|
By default, a mapping for the OAI parser has been added to map standard oai_dc fields to Hyrax basic_metadata. The other parsers have no default mapping, and will map any incoming fields to Hyrax properties with the same name. Configurations can be added in `config/initializers/bulkrax.rb`
|
124
126
|
|
125
|
-
Configuring field mappings is documented in the [Bulkrax Configuration Guide](https://github.com/samvera
|
127
|
+
Configuring field mappings is documented in the [Bulkrax Configuration Guide](https://github.com/samvera/bulkrax/wiki/Configuring-Bulkrax).
|
126
128
|
|
127
129
|
## Importing Files
|
128
130
|
|
@@ -153,7 +155,7 @@ end
|
|
153
155
|
|
154
156
|
## Customizing Bulkrax
|
155
157
|
|
156
|
-
For further information on how to extend and customize Bulkrax, please see the [Bulkrax Customization Guide](https://github.com/samvera
|
158
|
+
For further information on how to extend and customize Bulkrax, please see the [Bulkrax Customization Guide](https://github.com/samvera/bulkrax/wiki/Customizing-Bulkrax).
|
157
159
|
|
158
160
|
## How it Works
|
159
161
|
Once you have Bulkrax installed, you will have access to an easy to use interface with which you are able to create, edit, delete, run, and re-run imports and exports.
|
@@ -180,16 +182,21 @@ To delete an importer or exporter, select the delete (x) icon.
|
|
180
182
|
### Downloading an export
|
181
183
|
Once your the exporter has run, a download icon will appear on the exporters menu page.
|
182
184
|
|
185
|
+
## Compatibility
|
186
|
+
|
187
|
+
* Ruby 2.7 or newer is required
|
188
|
+
* Hyrax 2.3 or newer is required
|
189
|
+
|
183
190
|
## Contributing
|
184
191
|
If you're working on a PR for this project, create a feature branch off of `main`.
|
185
192
|
|
186
193
|
This repository follows the [Samvera Community Code of Conduct](https://samvera.atlassian.net/wiki/spaces/samvera/pages/405212316/Code+of+Conduct) and [language recommendations](https://github.com/samvera/maintenance/blob/master/templates/CONTRIBUTING.md#language). Please ***do not*** create a branch called `master` for this repository or as part of your pull request; the branch will either need to be removed or renamed before it can be considered for inclusion in the code base and history of this repository.
|
187
194
|
|
188
195
|
See
|
189
|
-
[CONTRIBUTING.md](https://github.com/samvera
|
196
|
+
[CONTRIBUTING.md](https://github.com/samvera/bulkrax/blob/main/CONTRIBUTING.md)
|
190
197
|
for contributing guidelines.
|
191
198
|
|
192
|
-
We encourage everyone to help improve this project. Bug reports and pull requests are welcome on GitHub at https://github.com/samvera
|
199
|
+
We encourage everyone to help improve this project. Bug reports and pull requests are welcome on GitHub at https://github.com/samvera/bulkrax.
|
193
200
|
|
194
201
|
This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](https://contributor-covenant.org) code of conduct.
|
195
202
|
|
@@ -14,7 +14,7 @@ function hideUnhide(field) {
|
|
14
14
|
}
|
15
15
|
};
|
16
16
|
|
17
|
-
function addRequired(selectedSource) {
|
17
|
+
function addRequired(selectedSource) {
|
18
18
|
selectedSource.addClass('required').attr('required', 'required');
|
19
19
|
selectedSource.parent().addClass('required');
|
20
20
|
}
|
@@ -26,14 +26,14 @@ function removeRequired(allSources) {
|
|
26
26
|
|
27
27
|
// hide all export_source
|
28
28
|
function hide(allSources) {
|
29
|
-
allSources.addClass('
|
30
|
-
allSources.find('#exporter_export_source').addClass('
|
29
|
+
allSources.addClass('d-none');
|
30
|
+
allSources.find('#exporter_export_source').addClass('.d-none').attr('type', 'd-none');
|
31
31
|
}
|
32
32
|
|
33
33
|
// unhide selected export_source
|
34
34
|
function unhideSelected(selectedSource) {
|
35
|
-
selectedSource.removeClass('
|
36
|
-
selectedSource.parent().removeClass('
|
35
|
+
selectedSource.removeClass('d-none').removeAttr('type');
|
36
|
+
selectedSource.parent().removeClass('d-none').removeAttr('type');
|
37
37
|
};
|
38
38
|
|
39
39
|
// add the autocomplete javascript
|
@@ -1,9 +1,26 @@
|
|
1
1
|
<% unless defined?(::Hyku) %>
|
2
2
|
// enables the tabs in the importers/exporters pages.
|
3
3
|
$(document).ready(function() {
|
4
|
-
$('.nav-tabs a').click(function
|
4
|
+
$('.bulkrax-nav-tab-top-margin.nav-tabs a').click(function(e) {
|
5
5
|
e.preventDefault();
|
6
|
-
|
6
|
+
|
7
|
+
// Remove active class from all tabs and hide all tab content
|
8
|
+
$('.bulkrax-nav-tab-top-margin.nav-tabs a').parent().removeClass('active');
|
9
|
+
$('.tab-content .tab-pane').removeClass('active');
|
10
|
+
|
11
|
+
// Add active class to clicked tab and show its content
|
12
|
+
$(this).parent().addClass('active');
|
13
|
+
$($(this).attr('href')).addClass('active');
|
7
14
|
});
|
15
|
+
|
16
|
+
$('#full-errors-tab, #full-errors-tab a').click(function(e) {
|
17
|
+
$('#raw-errors-tab, #bulkrax-raw-toggle-1').removeClass('active');
|
18
|
+
$('#full-errors-tab, #bulkrax-full-toggle-1').addClass('active');
|
19
|
+
})
|
20
|
+
|
21
|
+
$('#raw-errors-tab, #raw-errors-tab a').click(function(e) {
|
22
|
+
$('#full-errors-tab, #bulkrax-full-toggle-1').removeClass('active');
|
23
|
+
$('#raw-errors-tab, #bulkrax-raw-toggle-1').addClass('active');
|
24
|
+
})
|
8
25
|
});
|
9
26
|
<% end %>
|
@@ -103,7 +103,11 @@ module Bulkrax
|
|
103
103
|
end
|
104
104
|
|
105
105
|
def search_by_identifier
|
106
|
-
work_index
|
106
|
+
# TODO(alishaevn): return the proper `work_index` value below
|
107
|
+
# ref: https://github.com/samvera-labs/bulkrax/issues/866
|
108
|
+
# ref:https://github.com/samvera-labs/bulkrax/issues/867
|
109
|
+
# work_index = ::ActiveFedora.index_field_mapper.solr_name(work_identifier, :facetable)
|
110
|
+
work_index = work_identifier
|
107
111
|
query = { work_index =>
|
108
112
|
source_identifier_value }
|
109
113
|
# Query can return partial matches (something6 matches both something6 and something68)
|
@@ -119,7 +123,7 @@ module Bulkrax
|
|
119
123
|
def create
|
120
124
|
attrs = transform_attributes
|
121
125
|
@object = klass.new
|
122
|
-
object.reindex_extent = Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX if object.respond_to?(:reindex_extent)
|
126
|
+
object.reindex_extent = Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX if defined?(Hyrax::Adapters::NestingIndexAdapter) && object.respond_to?(:reindex_extent)
|
123
127
|
run_callbacks :save do
|
124
128
|
run_callbacks :create do
|
125
129
|
if klass == Collection
|
@@ -249,7 +253,7 @@ module Bulkrax
|
|
249
253
|
def collection_type(attrs)
|
250
254
|
return attrs if attrs['collection_type_gid'].present?
|
251
255
|
|
252
|
-
attrs['collection_type_gid'] = Hyrax::CollectionType.find_or_create_default_collection_type.
|
256
|
+
attrs['collection_type_gid'] = Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s
|
253
257
|
attrs
|
254
258
|
end
|
255
259
|
|
@@ -98,7 +98,7 @@ module Bulkrax
|
|
98
98
|
|
99
99
|
if errors.present?
|
100
100
|
# rubocop:disable Rails/SkipsModelValidations
|
101
|
-
|
101
|
+
ImporterRun.update_counters(importer_run_id, failed_relationships: number_of_failures)
|
102
102
|
# rubocop:enable Rails/SkipsModelValidations
|
103
103
|
|
104
104
|
parent_entry&.set_status_info(errors.last, importer_run)
|
@@ -108,7 +108,7 @@ module Bulkrax
|
|
108
108
|
return false # stop current job from continuing to run after rescheduling
|
109
109
|
else
|
110
110
|
# rubocop:disable Rails/SkipsModelValidations
|
111
|
-
|
111
|
+
ImporterRun.update_counters(importer_run_id, processed_relationships: number_of_successes)
|
112
112
|
# rubocop:enable Rails/SkipsModelValidations
|
113
113
|
end
|
114
114
|
end
|
@@ -158,7 +158,8 @@ module Bulkrax
|
|
158
158
|
end
|
159
159
|
|
160
160
|
def add_to_collection(child_record, parent_record)
|
161
|
-
parent_record.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX)
|
161
|
+
parent_record.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
162
|
+
defined?(Hyrax::Adapters::NestingIndexAdapter)
|
162
163
|
child_record.member_of_collections << parent_record
|
163
164
|
child_record.save!
|
164
165
|
end
|
@@ -4,17 +4,17 @@ module Bulkrax
|
|
4
4
|
class DeleteJob < ApplicationJob
|
5
5
|
queue_as :import
|
6
6
|
|
7
|
-
# rubocop:disable Rails/SkipsModelValidations
|
8
7
|
def perform(entry, importer_run)
|
9
8
|
obj = entry.factory.find
|
10
9
|
obj&.delete
|
11
|
-
|
12
|
-
ImporterRun.
|
10
|
+
# rubocop:disable Rails/SkipsModelValidations
|
11
|
+
ImporterRun.increment_counter(:deleted_records, importer_run.id)
|
12
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run.id)
|
13
|
+
# rubocop:enable Rails/SkipsModelValidations
|
13
14
|
entry.save!
|
14
15
|
entry.importer.current_run = ImporterRun.find(importer_run.id)
|
15
16
|
entry.importer.record_status
|
16
17
|
entry.set_status_info("Deleted", ImporterRun.find(importer_run.id))
|
17
18
|
end
|
18
|
-
# rubocop:enable Rails/SkipsModelValidations
|
19
19
|
end
|
20
20
|
end
|
@@ -12,17 +12,17 @@ module Bulkrax
|
|
12
12
|
entry.save
|
13
13
|
rescue StandardError
|
14
14
|
# rubocop:disable Rails/SkipsModelValidations
|
15
|
-
|
16
|
-
|
15
|
+
ExporterRun.increment_counter(:failed_records, args[1])
|
16
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
17
17
|
raise
|
18
18
|
else
|
19
19
|
if entry.failed?
|
20
|
-
|
21
|
-
|
20
|
+
ExporterRun.increment_counter(:failed_records, args[1])
|
21
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
22
22
|
raise entry.reload.current_status.error_class.constantize
|
23
23
|
else
|
24
|
-
|
25
|
-
|
24
|
+
ExporterRun.increment_counter(:processed_records, args[1])
|
25
|
+
ExporterRun.decrement_counter(:enqueued_records, args[1]) unless exporter_run.reload.enqueued_records <= 0
|
26
26
|
end
|
27
27
|
# rubocop:enable Rails/SkipsModelValidations
|
28
28
|
end
|
@@ -10,13 +10,13 @@ module Bulkrax
|
|
10
10
|
begin
|
11
11
|
entry.build
|
12
12
|
entry.save!
|
13
|
-
ImporterRun.
|
14
|
-
ImporterRun.
|
15
|
-
ImporterRun.
|
13
|
+
ImporterRun.increment_counter(:processed_records, args[1])
|
14
|
+
ImporterRun.increment_counter(:processed_collections, args[1])
|
15
|
+
ImporterRun.decrement_counter(:enqueued_records, args[1]) unless ImporterRun.find(args[1]).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
16
16
|
rescue => e
|
17
|
-
ImporterRun.
|
18
|
-
ImporterRun.
|
19
|
-
ImporterRun.
|
17
|
+
ImporterRun.increment_counter(:failed_records, args[1])
|
18
|
+
ImporterRun.increment_counter(:failed_collections, args[1])
|
19
|
+
ImporterRun.decrement_counter(:enqueued_records, args[1]) unless ImporterRun.find(args[1]).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
20
20
|
raise e
|
21
21
|
end
|
22
22
|
entry.importer.current_run = ImporterRun.find(args[1])
|
@@ -21,14 +21,14 @@ module Bulkrax
|
|
21
21
|
entry.build
|
22
22
|
if entry.succeeded?
|
23
23
|
# rubocop:disable Rails/SkipsModelValidations
|
24
|
-
ImporterRun.
|
25
|
-
ImporterRun.
|
24
|
+
ImporterRun.increment_counter(:processed_records, importer_run_id)
|
25
|
+
ImporterRun.increment_counter(:processed_file_sets, importer_run_id)
|
26
26
|
else
|
27
|
-
ImporterRun.
|
28
|
-
ImporterRun.
|
27
|
+
ImporterRun.increment_counter(:failed_records, importer_run_id)
|
28
|
+
ImporterRun.increment_counter(:failed_file_sets, importer_run_id)
|
29
29
|
# rubocop:enable Rails/SkipsModelValidations
|
30
30
|
end
|
31
|
-
ImporterRun.
|
31
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run_id) unless ImporterRun.find(importer_run_id).enqueued_records <= 0 # rubocop:disable Rails/SkipsModelValidations
|
32
32
|
entry.save!
|
33
33
|
entry.importer.current_run = ImporterRun.find(importer_run_id)
|
34
34
|
entry.importer.record_status
|
@@ -40,7 +40,7 @@ module Bulkrax
|
|
40
40
|
if entry.import_attempts < 5
|
41
41
|
ImportFileSetJob.set(wait: (entry.import_attempts + 1).minutes).perform_later(entry_id, importer_run_id)
|
42
42
|
else
|
43
|
-
ImporterRun.
|
43
|
+
ImporterRun.decrement_counter(:enqueued_records, importer_run_id) # rubocop:disable Rails/SkipsModelValidations
|
44
44
|
entry.set_status_info(e)
|
45
45
|
end
|
46
46
|
end
|
@@ -23,16 +23,16 @@ module Bulkrax
|
|
23
23
|
entry = Entry.find(entry_id)
|
24
24
|
entry.build
|
25
25
|
if entry.status == "Complete"
|
26
|
-
ImporterRun.
|
27
|
-
ImporterRun.
|
26
|
+
ImporterRun.increment_counter(:processed_records, run_id)
|
27
|
+
ImporterRun.increment_counter(:processed_works, run_id)
|
28
28
|
else
|
29
29
|
# do not retry here because whatever parse error kept you from creating a work will likely
|
30
30
|
# keep preventing you from doing so.
|
31
|
-
ImporterRun.
|
32
|
-
ImporterRun.
|
31
|
+
ImporterRun.increment_counter(:failed_records, run_id)
|
32
|
+
ImporterRun.increment_counter(:failed_works, run_id)
|
33
33
|
end
|
34
34
|
# Regardless of completion or not, we want to decrement the enqueued records.
|
35
|
-
ImporterRun.
|
35
|
+
ImporterRun.decrement_counter(:enqueued_records, run_id) unless ImporterRun.find(run_id).enqueued_records <= 0
|
36
36
|
|
37
37
|
entry.save!
|
38
38
|
entry.importer.current_run = ImporterRun.find(run_id)
|
@@ -15,7 +15,7 @@ module Bulkrax
|
|
15
15
|
def add_collection_type_gid
|
16
16
|
return if self.parsed_metadata['collection_type_gid'].present?
|
17
17
|
|
18
|
-
self.parsed_metadata['collection_type_gid'] = ::Hyrax::CollectionType.find_or_create_default_collection_type.
|
18
|
+
self.parsed_metadata['collection_type_gid'] = ::Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s
|
19
19
|
end
|
20
20
|
end
|
21
21
|
end
|
@@ -88,8 +88,10 @@ module Bulkrax
|
|
88
88
|
|
89
89
|
def validate_record
|
90
90
|
raise StandardError, 'Record not found' if record.nil?
|
91
|
-
|
92
|
-
|
91
|
+
unless importerexporter.parser.required_elements?(record)
|
92
|
+
raise StandardError, "Missing required elements, missing element(s) are: "\
|
93
|
+
"#{importerexporter.parser.missing_elements(record).join(', ')}"
|
94
|
+
end
|
93
95
|
end
|
94
96
|
|
95
97
|
def add_identifier
|
@@ -270,8 +270,8 @@ module Bulkrax
|
|
270
270
|
current_run.invalid_records ||= ""
|
271
271
|
current_run.invalid_records += message
|
272
272
|
current_run.save
|
273
|
-
ImporterRun.
|
274
|
-
ImporterRun.
|
273
|
+
ImporterRun.increment_counter(:failed_records, current_run.id)
|
274
|
+
ImporterRun.decrement_counter(:enqueued_records, current_run.id) unless ImporterRun.find(current_run.id).enqueued_records <= 0 # rubocop:disable Style/IdenticalConditionalBranches
|
275
275
|
end
|
276
276
|
# rubocop:enable Rails/SkipsModelValidations
|
277
277
|
|
@@ -33,9 +33,10 @@ module Bulkrax
|
|
33
33
|
model_field_mappings.map(&:to_sym).each do |model_mapping|
|
34
34
|
next unless r.key?(model_mapping)
|
35
35
|
|
36
|
-
|
36
|
+
model = r[model_mapping].nil? ? "" : r[model_mapping].strip
|
37
|
+
if model.casecmp('collection').zero?
|
37
38
|
@collections << r
|
38
|
-
elsif
|
39
|
+
elsif model.casecmp('fileset').zero?
|
39
40
|
@file_sets << r
|
40
41
|
else
|
41
42
|
@works << r
|
@@ -67,7 +67,7 @@ module Bulkrax
|
|
67
67
|
metadata = {
|
68
68
|
visibility: 'open'
|
69
69
|
}
|
70
|
-
metadata[:collection_type_gid] = Hyrax::CollectionType.find_or_create_default_collection_type.
|
70
|
+
metadata[:collection_type_gid] = Hyrax::CollectionType.find_or_create_default_collection_type.to_global_id.to_s if defined?(::Hyrax)
|
71
71
|
|
72
72
|
collections.each_with_index do |set, index|
|
73
73
|
next unless collection_name == 'all' || collection_name == set.spec
|
@@ -63,7 +63,8 @@ module Bulkrax
|
|
63
63
|
remove_relationships_from_work(obj)
|
64
64
|
end
|
65
65
|
|
66
|
-
obj.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
66
|
+
obj.try(:reindex_extent=, Hyrax::Adapters::NestingIndexAdapter::LIMITED_REINDEX) if
|
67
|
+
defined?(Hyrax::Adapters::NestingIndexAdapter)
|
67
68
|
obj.save!
|
68
69
|
end
|
69
70
|
end
|
@@ -33,8 +33,8 @@
|
|
33
33
|
label: t('bulkrax.exporter.labels.importer'),
|
34
34
|
required: true,
|
35
35
|
prompt: 'Select from the list',
|
36
|
-
label_html: { class: 'importer export-source-option
|
37
|
-
input_html: { class: 'importer export-source-option
|
36
|
+
label_html: { class: 'importer export-source-option d-none' },
|
37
|
+
input_html: { class: 'importer export-source-option d-none form-control' },
|
38
38
|
collection: form.object.importers_list.sort %>
|
39
39
|
|
40
40
|
<%= form.input :export_source_collection,
|
@@ -42,9 +42,9 @@
|
|
42
42
|
label: t('bulkrax.exporter.labels.collection'),
|
43
43
|
required: true,
|
44
44
|
placeholder: @collection&.title&.first,
|
45
|
-
label_html: { class: 'collection export-source-option
|
45
|
+
label_html: { class: 'collection export-source-option d-none' },
|
46
46
|
input_html: {
|
47
|
-
class: 'collection export-source-option
|
47
|
+
class: 'collection export-source-option d-none form-control',
|
48
48
|
data: {
|
49
49
|
'autocomplete-url' => '/authorities/search/collections',
|
50
50
|
'autocomplete' => 'collection'
|
@@ -56,8 +56,8 @@
|
|
56
56
|
label: t('bulkrax.exporter.labels.worktype'),
|
57
57
|
required: true,
|
58
58
|
prompt: 'Select from the list',
|
59
|
-
label_html: { class: 'worktype export-source-option
|
60
|
-
input_html: { class: 'worktype export-source-option
|
59
|
+
label_html: { class: 'worktype export-source-option d-none' },
|
60
|
+
input_html: { class: 'worktype export-source-option d-none form-control' },
|
61
61
|
collection: Bulkrax.curation_concerns.map { |cc| [cc.to_s, cc.to_s] } %>
|
62
62
|
|
63
63
|
<%= form.input :limit,
|
@@ -80,7 +80,7 @@
|
|
80
80
|
as: :boolean,
|
81
81
|
label: t('bulkrax.exporter.labels.filter_by_date') %>
|
82
82
|
|
83
|
-
<div id="date_filter_picker" class="
|
83
|
+
<div id="date_filter_picker" class="d-none">
|
84
84
|
<%= form.input :start_date,
|
85
85
|
as: :date,
|
86
86
|
label: t('bulkrax.exporter.labels.start_date'),
|
@@ -136,13 +136,13 @@
|
|
136
136
|
// get the date filter option and show the corresponding date selectors
|
137
137
|
$('.exporter_date_filter').change(function () {
|
138
138
|
if ($('.exporter_date_filter').find(".boolean").is(":checked"))
|
139
|
-
$('#date_filter_picker').removeClass('
|
139
|
+
$('#date_filter_picker').removeClass('d-none');
|
140
140
|
else
|
141
|
-
$('#date_filter_picker').addClass('
|
141
|
+
$('#date_filter_picker').addClass('d-none');
|
142
142
|
});
|
143
143
|
|
144
144
|
if ($('.exporter_date_filter').find(".boolean").is(":checked"))
|
145
|
-
$('#date_filter_picker').removeClass('
|
145
|
+
$('#date_filter_picker').removeClass('d-none');
|
146
146
|
});
|
147
147
|
});
|
148
148
|
</script>
|
@@ -96,9 +96,9 @@
|
|
96
96
|
<div class="bulkrax-nav-tab-bottom-margin">
|
97
97
|
<!-- Nav tabs -->
|
98
98
|
<ul class="bulkrax-nav-tab-top-margin tab-nav nav nav-tabs" role="tablist">
|
99
|
-
<li role="presentation" class=
|
100
|
-
<li role="presentation"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.collection_entries') %></a></li>
|
101
|
-
<li role="presentation"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.file_set_entries') %></a></li>
|
99
|
+
<li role="presentation" class="nav-link active"><a href="#work-entries" aria-controls="work-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.work_entries') %></a></li>
|
100
|
+
<li role="presentation" class="nav-link"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.collection_entries') %></a></li>
|
101
|
+
<li role="presentation" class="nav-link"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.exporter.labels.file_set_entries') %></a></li>
|
102
102
|
</ul>
|
103
103
|
<!-- Tab panes -->
|
104
104
|
<div class="tab-content outline">
|
@@ -76,9 +76,9 @@
|
|
76
76
|
<div class="bulkrax-nav-tab-bottom-margin">
|
77
77
|
<!-- Nav tabs -->
|
78
78
|
<ul class="bulkrax-nav-tab-top-margin tab-nav nav nav-tabs" role="tablist">
|
79
|
-
<li role="presentation" class=
|
80
|
-
<li role="presentation"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.collection_entries') %></a></li>
|
81
|
-
<li role="presentation"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.file_set_entries') %></a></li>
|
79
|
+
<li role="presentation" class="nav-link active"><a href="#work-entries" aria-controls="work-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.work_entries') %></a></li>
|
80
|
+
<li role="presentation" class="nav-link"><a href="#collection-entries" aria-controls="collection-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.collection_entries') %></a></li>
|
81
|
+
<li role="presentation" class="nav-link"><a href="#file-set-entries" aria-controls="file-set-entries" role="tab" data-toggle="tab"><%= t('bulkrax.importer.labels.file_set_entries') %></a></li>
|
82
82
|
</ul>
|
83
83
|
<!-- Tab panes -->
|
84
84
|
<div class="tab-content outline">
|
@@ -13,8 +13,8 @@
|
|
13
13
|
<div class="bulkrax-nav-tab-bottom-margin">
|
14
14
|
<!-- Toggle buttons -->
|
15
15
|
<div class="btn-group pull-right" role="group" aria-label="...">
|
16
|
-
<button type="button" class="btn btn-default active"><a href="#bulkrax-full-toggle-1" aria-controls="bulkrax-full-toggle-1" role="tab" data-toggle="tab">Full</a></button>
|
17
|
-
<button type="button" class="btn btn-default"><a href="#bulkrax-raw-toggle-1" aria-controls="bulkrax-raw-toggle-1" role="tab" data-toggle="tab">Raw</a></button>
|
16
|
+
<button id="full-errors-tab" type="button" class="btn btn-default active"><a href="#bulkrax-full-toggle-1" aria-controls="bulkrax-full-toggle-1" role="tab" data-toggle="tab">Full</a></button>
|
17
|
+
<button id="raw-errors-tab" type="button" class="btn btn-default"><a href="#bulkrax-raw-toggle-1" aria-controls="bulkrax-raw-toggle-1" role="tab" data-toggle="tab">Raw</a></button>
|
18
18
|
</div>
|
19
19
|
<!-- Tab panes -->
|
20
20
|
<div class="tab-content">
|
@@ -1,12 +1,20 @@
|
|
1
1
|
<% if current_ability.can_import_works? %>
|
2
|
-
<%= menu.nav_link(
|
3
|
-
|
2
|
+
<%= menu.nav_link(
|
3
|
+
bulkrax.importers_path,
|
4
|
+
title: t('bulkrax.admin.sidebar.importers'),
|
5
|
+
class: "nav-link",
|
6
|
+
onclick: "dontChangeAccordion(event);"
|
7
|
+
) do %>
|
4
8
|
<span class="fa fa-cloud-upload" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.importers') %></span>
|
5
9
|
<% end %>
|
6
10
|
<% end %>
|
7
11
|
<% if current_ability.can_export_works? %>
|
8
|
-
<%= menu.nav_link(
|
9
|
-
|
12
|
+
<%= menu.nav_link(
|
13
|
+
bulkrax.exporters_path,
|
14
|
+
title: t('bulkrax.admin.sidebar.exporters'),
|
15
|
+
class: "nav-link",
|
16
|
+
onclick: "dontChangeAccordion(event);"
|
17
|
+
) do %>
|
10
18
|
<span class="fa fa-cloud-download" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('bulkrax.admin.sidebar.exporters') %></span>
|
11
19
|
<% end %>
|
12
20
|
<% end %>
|
@@ -6,11 +6,13 @@ end
|
|
6
6
|
|
7
7
|
class ChangeImporterAndExporterToPolymorphic < ActiveRecord::Migration[5.1]
|
8
8
|
def change
|
9
|
-
|
10
|
-
remove_foreign_key :bulkrax_entries, column: :importer_id
|
11
|
-
|
12
|
-
|
9
|
+
begin # deal with odd bug around foreign keys in ci
|
10
|
+
remove_foreign_key :bulkrax_entries, column: :importer_id if foreign_key_exists?(:bulkrax_entries, column: :importer_id)
|
11
|
+
rescue ArgumentError
|
12
|
+
# do nothing
|
13
13
|
end
|
14
|
+
remove_index :bulkrax_entries, :importer_id if index_exists?(:bulkrax_entries, :importer_id)
|
15
|
+
rename_column :bulkrax_entries, :importer_id, :importerexporter_id if column_exists?(:bulkrax_entries, :importer_id)
|
14
16
|
add_column :bulkrax_entries, :importerexporter_type, :string, after: :id, default: 'Bulkrax::Importer' unless column_exists?(:bulkrax_entries, :importerexporter_type)
|
15
17
|
end
|
16
18
|
end
|
@@ -10,13 +10,17 @@ module Bulkrax
|
|
10
10
|
#
|
11
11
|
# This module came about through a desire to expose a quick means of vetting the accuracy of the
|
12
12
|
# different parsers.
|
13
|
+
#
|
14
|
+
# @see .entry_for
|
13
15
|
module EntrySpecHelper
|
14
16
|
##
|
15
17
|
# @api public
|
16
18
|
# @since v5.0.1
|
17
19
|
#
|
18
|
-
# The purpose of this method is encapsulate the logic of creating the appropriate
|
19
|
-
# object based on the given data, identifier, and parser_class_name.
|
20
|
+
# The purpose of this method is encapsulate the logic of creating the appropriate
|
21
|
+
# {Bulkrax::Entry} object based on the given data, identifier, and parser_class_name. Due to
|
22
|
+
# the different means of instantiation of {Bulkrax::Entry} subclasses, there are several
|
23
|
+
# optional parameters.
|
20
24
|
#
|
21
25
|
# From that entry, you should be able to test how {Bulkrax::Entry#build_metadata} populates the
|
22
26
|
# {Bulkrax::Entry#parsed_metadata} variable. Other uses may emerge.
|
@@ -29,9 +33,47 @@ module Bulkrax
|
|
29
33
|
# @param parser_class_name [String] The name of the parser class you're wanting to test.
|
30
34
|
# @param type [Sybmol] The type of entry (e.g. :entry, :collection, :file_set) for testing.
|
31
35
|
# @param options [Hash<Symbol,Object>] these are to be passed along into the instantiation of
|
32
|
-
# the various classes.
|
36
|
+
# the various classes.
|
37
|
+
# @option options [String] importer_name (Optional) The name of the test importer. One will be
|
38
|
+
# auto-assigned if unprovided.
|
39
|
+
# @option options [String] importer_admin_set_id (Optional) The ID of an admin set to deposit
|
40
|
+
# into. One will be auto-assigned if unprovided. And this admin set does not need to
|
41
|
+
# be persisted nor exist. It is simply a required parameter for instantiating an
|
42
|
+
# importer.
|
43
|
+
# @option options [User] user (Optional) The user who is performing the import. One will be
|
44
|
+
# auto-assigned if unprovided. The user does not need to be persisted. It is simply a
|
45
|
+
# required parameter for instantiating an importer
|
46
|
+
# @option options [Integer] limit (Optional) You really shouldn't need to set this, but for
|
47
|
+
# completeness it is provided.
|
48
|
+
# @option options [Hash<String, Object>] importer_field_mappings Each parser class may require
|
49
|
+
# different field mappings. See the given examples for more details.
|
50
|
+
#
|
51
|
+
# @return [Bulkrax::Entry] a subclass of {Bulkrax::Entry} based on the application's
|
52
|
+
# configuration. It would behoove you to write a spec regarding the returned entry's
|
53
|
+
# class.
|
54
|
+
#
|
55
|
+
# @example
|
56
|
+
# entry = Bulkrax::EntrySpecHelper.entry_for(
|
57
|
+
# data: { source_identifier: "123", title: "Hello World" },
|
58
|
+
# parser_class_name: "Bulkrax::CsvParser",
|
59
|
+
# importer_field_mappings: { 'import_file_path' => "path/to/file.csv" }
|
60
|
+
# )
|
61
|
+
#
|
62
|
+
# @note In the case of the Bulkrax::CsvParser, the :data keyword is a Hash, where the keys are
|
63
|
+
# the column name of the CSV you're importing. The 'import_file_path' is a path to a CSV
|
64
|
+
# file. That CSV's columns does not need to match the :data's keys, though there may be
|
65
|
+
# required headers on that CSV based on the parser implementation.
|
33
66
|
#
|
34
|
-
# @
|
67
|
+
# @example
|
68
|
+
# entry = Bulkrax::EntrySpecHelper.entry_for(
|
69
|
+
# identifier: identifier,
|
70
|
+
# data: File.read("/path/to/some/file.xml"),
|
71
|
+
# parser_class_name: "Bulkrax::OaiDcParser",
|
72
|
+
# parser_fields: { "base_url" => "http://oai.adventistdigitallibrary.org/OAI-script" }
|
73
|
+
# )
|
74
|
+
#
|
75
|
+
# @note In the case of an OaiParser, the :data keyword should be a String. And you'll need to
|
76
|
+
# provide a :parser_fields with a "base_url".
|
35
77
|
def self.entry_for(data:, identifier:, parser_class_name:, type: :entry, **options)
|
36
78
|
importer = importer_for(parser_class_name: parser_class_name, **options)
|
37
79
|
entry_type_method_name = ENTRY_TYPE_TO_METHOD_NAME_MAP.fetch(type)
|
@@ -58,6 +100,7 @@ module Bulkrax
|
|
58
100
|
**options)
|
59
101
|
end
|
60
102
|
|
103
|
+
##
|
61
104
|
# @api public
|
62
105
|
#
|
63
106
|
# @param parser_class_name [String]
|
@@ -121,16 +164,6 @@ module Bulkrax
|
|
121
164
|
end
|
122
165
|
private_class_method :importer_for
|
123
166
|
|
124
|
-
##
|
125
|
-
# @api private
|
126
|
-
#
|
127
|
-
# @param data [Hash<Symbol,String>] we're expecting a hash with keys that are symbols and then
|
128
|
-
# values that are strings.
|
129
|
-
#
|
130
|
-
# @return [Bulkrax::CsvEntry]
|
131
|
-
#
|
132
|
-
# @note As a foible of this implementation, you'll need to include along a CSV to establish the
|
133
|
-
# columns that you'll parse (e.g. the first row
|
134
167
|
def self.build_csv_entry_for(importer:, data:, identifier:, entry_class:, **_options)
|
135
168
|
entry_class.new(
|
136
169
|
importerexporter: importer,
|
@@ -138,13 +171,8 @@ module Bulkrax
|
|
138
171
|
raw_metadata: data
|
139
172
|
)
|
140
173
|
end
|
174
|
+
private_class_method :build_csv_entry_for
|
141
175
|
|
142
|
-
##
|
143
|
-
# @api private
|
144
|
-
#
|
145
|
-
# @param data [String] we're expecting a string that is well-formed XML for OAI parsing.
|
146
|
-
#
|
147
|
-
# @return [Bulkrax::OaiEntry]
|
148
176
|
def self.build_oai_entry_for(importer:, data:, identifier:, entry_class:, **options)
|
149
177
|
# The raw record assumes we take the XML data, parse it and then send that to the
|
150
178
|
# OAI::GetRecordResponse object.
|
@@ -165,13 +193,8 @@ module Bulkrax
|
|
165
193
|
raw_metadata: raw_metadata
|
166
194
|
)
|
167
195
|
end
|
196
|
+
private_class_method :build_oai_entry_for
|
168
197
|
|
169
|
-
##
|
170
|
-
# @api private
|
171
|
-
#
|
172
|
-
# @param data [String] we're expecting a string that is well-formed XML.
|
173
|
-
#
|
174
|
-
# @return [Bulkrax::XmlEntry]
|
175
198
|
def self.build_xml_entry_for(importer:, data:, identifier:, entry_class:, **options)
|
176
199
|
raw_metadata = {
|
177
200
|
importer.parser.source_identifier.to_s => identifier,
|
@@ -186,5 +209,6 @@ module Bulkrax
|
|
186
209
|
raw_metadata: raw_metadata
|
187
210
|
)
|
188
211
|
end
|
212
|
+
private_class_method :build_xml_entry_for
|
189
213
|
end
|
190
214
|
end
|
data/lib/bulkrax/version.rb
CHANGED
@@ -80,6 +80,7 @@ Bulkrax.setup do |config|
|
|
80
80
|
end
|
81
81
|
|
82
82
|
# Sidebar for hyrax 3+ support
|
83
|
+
# rubocop:disable Style/IfUnlessModifier
|
83
84
|
if Object.const_defined?(:Hyrax) && ::Hyrax::DashboardController&.respond_to?(:sidebar_partials)
|
84
85
|
Hyrax::DashboardController.sidebar_partials[:repository_content] << "hyrax/dashboard/sidebar/bulkrax_sidebar_additions"
|
85
86
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: bulkrax
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 5.
|
4
|
+
version: 5.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Rob Kaufman
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-11-29 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rails
|
@@ -52,20 +52,6 @@ dependencies:
|
|
52
52
|
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
-
- !ruby/object:Gem::Dependency
|
56
|
-
name: dry-monads
|
57
|
-
requirement: !ruby/object:Gem::Requirement
|
58
|
-
requirements:
|
59
|
-
- - "~>"
|
60
|
-
- !ruby/object:Gem::Version
|
61
|
-
version: 1.4.0
|
62
|
-
type: :runtime
|
63
|
-
prerelease: false
|
64
|
-
version_requirements: !ruby/object:Gem::Requirement
|
65
|
-
requirements:
|
66
|
-
- - "~>"
|
67
|
-
- !ruby/object:Gem::Version
|
68
|
-
version: 1.4.0
|
69
55
|
- !ruby/object:Gem::Dependency
|
70
56
|
name: iso8601
|
71
57
|
requirement: !ruby/object:Gem::Requirement
|