geoblacklight_sidecar_images 0.1.0 → 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (33) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +1 -0
  3. data/Gemfile +5 -5
  4. data/README.md +136 -11
  5. data/db/migrate/20180118203155_create_solr_document_sidecars.rb +4 -2
  6. data/db/migrate/20180118203519_create_sidecar_image_transitions.rb +29 -0
  7. data/geoblacklight_sidecar_images.gemspec +5 -3
  8. data/lib/generators/geoblacklight_sidecar_images/{uploaders_generator.rb → config_generator.rb} +4 -5
  9. data/lib/generators/geoblacklight_sidecar_images/install_generator.rb +8 -14
  10. data/lib/generators/geoblacklight_sidecar_images/models_generator.rb +24 -2
  11. data/lib/generators/geoblacklight_sidecar_images/templates/config/initializers/statesman.rb +5 -0
  12. data/lib/generators/geoblacklight_sidecar_images/templates/jobs/store_image_job.rb +10 -3
  13. data/lib/generators/geoblacklight_sidecar_images/templates/models/sidecar_image_state_machine.rb +19 -0
  14. data/lib/generators/geoblacklight_sidecar_images/templates/models/sidecar_image_transition.rb +7 -0
  15. data/lib/generators/geoblacklight_sidecar_images/templates/models/solr_document_sidecar.rb +34 -2
  16. data/lib/generators/geoblacklight_sidecar_images/templates/services/image_service.rb +103 -96
  17. data/lib/generators/geoblacklight_sidecar_images/templates/views/catalog/_index_split_default.html.erb +11 -1
  18. data/lib/geoblacklight_sidecar_images/version.rb +1 -1
  19. data/lib/tasks/geoblacklight_sidecar_images_tasks.rake +165 -39
  20. data/spec/jobs/store_image_job_spec.rb +16 -0
  21. data/spec/services/image_service_spec.rb +5 -8
  22. data/template.rb +3 -2
  23. metadata +48 -24
  24. data/lib/generators/geoblacklight_sidecar_images/assets_generator.rb +0 -31
  25. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-image.png +0 -0
  26. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-line.png +0 -0
  27. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-mixed.png +0 -0
  28. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-multipoint.png +0 -0
  29. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-paper-map.png +0 -0
  30. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-point.png +0 -0
  31. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-polygon.png +0 -0
  32. data/lib/generators/geoblacklight_sidecar_images/templates/assets/images/thumbnail-raster.png +0 -0
  33. data/lib/generators/geoblacklight_sidecar_images/templates/uploaders/image_uploader.rb +0 -55
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 3233e1aaff539c85ba41a37cd3e87b7a8ff083d7
4
- data.tar.gz: 330ef9966073b81e5fdc539287e25ef2333b98f8
3
+ metadata.gz: 520f1299deb657555a88fde5230b77fa5d37f79a
4
+ data.tar.gz: f9416c70247fc5a4c8354be1116f7570610c4959
5
5
  SHA512:
6
- metadata.gz: 5a8af424a1f10bec71d607c4997810b054aab5948b662a83cdf2325ed3005cf90d6dc0bd0b5a496a2a0303d454941b38b968e9c81c3fd8b3c56a97fc36ebfcf7
7
- data.tar.gz: d3fa62b61a58cbb07edd93ad94b1e93c2e652d9a06947a2011c9a40ffa30afd8949e33e3d68c3d7072f8c085bb3d82f794592c4f2984f6552e9b5711a2272542
6
+ metadata.gz: 888fef80d916c4469cef6412b64f083824bb4062f8dc53ec500424fac9b7ce90e8ff8e5db771bedf65d2c2ae8075d79b1b993ee254399d7954184e03caa06dd1
7
+ data.tar.gz: 49688593216ab742e2f97a851a5b57a5297d9123f38fde6d00f3d7ed96b081091e6526b5b78aec0fa5018cdac1df693df185b17ee96e29ec0792dd3abd1b3a19
data/.gitignore CHANGED
@@ -16,3 +16,4 @@ spec/test_app/yarn-error.log
16
16
  spec/test_app/storage/
17
17
  spec/test_app/tmp/
18
18
  .internal_test_app
19
+ .ruby-version
data/Gemfile CHANGED
@@ -37,11 +37,11 @@ if File.exist?(file)
37
37
  end
38
38
  else
39
39
  Bundler.ui.warn "[EngineCart] Unable to find test application dependencies in #{file}, using placeholder dependencies"
40
-
41
- gem 'aasm'
42
- gem 'carrierwave'
43
- gem 'geoblacklight', '~> 1.7'
40
+ gem 'geoblacklight', '~> 1.9'
44
41
  gem 'mini_magick'
45
- gem 'rails', '>= 4.2', '< 6'
42
+ gem 'image_processing', '~> 1.6'
43
+ gem 'mimemagic', '~> 0.3'
44
+ gem 'statesman', '~> 3.4'
45
+ gem 'rails', '~> 5.2.0', '< 6'
46
46
  end
47
47
  # END ENGINE_CART BLOCK
data/README.md CHANGED
@@ -8,16 +8,23 @@
8
8
  Store local copies of remote imagery in GeoBlacklight.
9
9
 
10
10
  ## Description
11
- This GeoBlacklight plugin helps capture remote images from geo web services and save them locally. It borrows the concept of a [SolrDocumentSidecar](https://github.com/projectblacklight/spotlight/blob/master/app/models/spotlight/solr_document_sidecar.rb) from [Spotlight](https://github.com/projectblacklight/spotlight), to have an ActiveRecord-based "sidecar" to match each non-AR SolrDocument. This allows us to use [carrierwave](https://github.com/carrierwaveuploader/carrierwave) to attach images to our documents.
11
+ This GeoBlacklight plugin captures remote images from geographic web services and saves them locally. It borrows the concept of a [SolrDocumentSidecar](https://github.com/projectblacklight/spotlight/blob/master/app/models/spotlight/solr_document_sidecar.rb) from [Spotlight](https://github.com/projectblacklight/spotlight), to have an ActiveRecord-based "sidecar" to match each non-AR SolrDocument. This allows us to use [ActiveStorage](https://github.com/rails/rails/tree/master/activestorage) to attach images to our solr documents.
12
12
 
13
13
  ### Example Screenshot
14
14
  ![Screenshot](screenshot.png)
15
15
 
16
16
  ## Requirements
17
17
 
18
+ * [Ruby on Rails 5.2](https://weblog.rubyonrails.org/releases/)
18
19
  * [GeoBlacklight](https://github.com/geoblacklight/geoblacklight)
19
20
  * [ImageMagick](https://github.com/ImageMagick/ImageMagick)
20
21
 
22
+ ## Suggested
23
+
24
+ * Background Job Processor
25
+
26
+ [Sidekiq](https://github.com/mperham/sidekiq) is an excellent choice if you need an opinion.
27
+
21
28
  ## Installation
22
29
 
23
30
  ### Existing GeoBlacklight Instance
@@ -52,21 +59,105 @@ $ rails new app-name -m https://raw.githubusercontent.com/ewlarson/geoblacklight
52
59
  ### Ingest Test Documents
53
60
 
54
61
  ```bash
55
- rake geoblacklight_sidecar_images:sample_data:ingest['<FULL_PATH_TO>/geoblacklight_sidecar_images/spec/fixtures/files']
62
+ # Run your GBL instance
63
+ rake geoblacklight:server
56
64
  ```
57
65
 
58
- ### Cache images
66
+ ```bash
67
+ rake gblsci:sample_data:ingest['<FULL_LOCAL_PATH_TO>/geoblacklight_sidecar_images/spec/fixtures/files']
68
+ ```
59
69
 
60
- #### All Thumbnails
70
+ ## Rake tasks
71
+
72
+ ### Harvest images
73
+
74
+ #### Harvest all images
75
+
76
+ Spawns background jobs to harvest images for all documents in your Solr index.
61
77
 
62
78
  ```bash
63
- rake geoblacklight_sidecar_images:images:precache_all
79
+ rake gblsci:images:harvest_all
64
80
  ```
65
81
 
66
- #### Individual Thumbnail
82
+ #### Harvest an individual image
83
+
84
+ Allows you to add images one document id at a time.
67
85
 
68
86
  ```bash
69
- rake geoblacklight_sidecar_images:images:precache_id['minnesota-iiif-jpg-83f4648a-125c-4000-a12f-aba2b432e7cd']
87
+ rake gblsci:images:harvest_doc_id['stanford-cz128vq0535']
88
+ ```
89
+
90
+ #### Harvest all incomplete states
91
+
92
+ Reattempt image harvesting for all non-successful state objects.
93
+
94
+ ```bash
95
+ rake gblsci:images:harvest_retry
96
+ ```
97
+
98
+ ### Check image states
99
+
100
+ ```bash
101
+ rake gblsci:images:harvest_states
102
+ ```
103
+
104
+ We use a state machine library to track success/failure of our harvest tasks. The states we track are:
105
+
106
+ * initialized - SolrDocumentSidecar created, no harvest attempt run
107
+ * queued - Harvest attempt queued as background job
108
+ * processing - Harvest attempt at work
109
+ * succeeded - Harvest was successful, image attached
110
+ * failed - Harvest failed, no image attached, error logged
111
+ * placeheld - Harvest was not successful, placeholder imagery will be used
112
+
113
+ ```ruby
114
+ SolrDocumentSidecar.image.attached? => false
115
+ SolrDocumentSidecar.image_state.current_state => "placeheld"
116
+ SolrDocumentSidecar.image_state.last_transition => #<SidecarImageTransition id: 207, to_state: "placeheld", metadata: {"solr_doc_id"=>"stanford-cg357zz0321", "solr_version"=>1616509329754554368, "placeheld"=>true, "viewer_protocol"=>"wms", "image_url"=>"http://geowebservices-restricted.stanford.edu/geoserver/wms/reflect?&FORMAT=image%2Fpng&TRANSPARENT=TRUE&LAYERS=druid:cg357zz0321&WIDTH=300&HEIGHT=300", "service_url"=>"http://geowebservices-restricted.stanford.edu/geoserver/wms/reflect?&FORMAT=image%2Fpng&TRANSPARENT=TRUE&LAYERS=druid:cg357zz0321&WIDTH=300&HEIGHT=300", "gblsi_thumbnail_uri"=>false, "error"=>"Faraday::Error::ConnectionFailed"},...>
117
+ ```
118
+
119
+ ### Destroy images
120
+
121
+ #### Remove everything
122
+
123
+ Remove all sidecar objects and attached images
124
+
125
+ ```bash
126
+ rake gblsci:images:harvest_purge_all
127
+ ```
128
+
129
+ #### Remove orphaned AR objects
130
+
131
+ Remove all sidecar objects and attached images for AR objects without a corresponding Solr document
132
+
133
+ ```bash
134
+ rake gblsci:images:harvest_purge_orphans
135
+ ```
136
+
137
+ #### Remove a batch
138
+
139
+ Remove sidecar objects and attached images via a CSV file of document ids
140
+
141
+ ```bash
142
+ rake gblsci:images:harvest_purge_orphans
143
+ ```
144
+
145
+ ### Troubleshooting
146
+
147
+ #### Harvest report
148
+
149
+ Generate a CSV file of sidecar objects and associated image state. Useful for debugging problem items.
150
+
151
+ ```bash
152
+ rake gblsci:images:harvest_report
153
+ ```
154
+
155
+ #### Failed state inspect
156
+
157
+ Prints details for failed state harvest objects to stdout
158
+
159
+ ```bash
160
+ rake gblsci:images:harvest_failed_state_inspect
70
161
  ```
71
162
 
72
163
  ## Prioritize Solr Thumbnail Field URIs
@@ -88,20 +179,54 @@ If you add a thumbnail uri to your geoblacklight solr documents...
88
179
 
89
180
  Then you can edit your GeoBlacklight settings.yml file to point at that solr field (Settings.GBLSI_THUMBNAIL_FIELD). Any docs in your index that have a value for that field will harvest the image at that URI instead of trying to retrieve an image via IIIF or the other web services.
90
181
 
182
+ ## View customization
183
+
184
+ This GBL plugin includes a custom catalog/_index_split_default.html.erb file. Look there for examples on calling the image method.
185
+
186
+ ```ruby
187
+ # Is there an image?
188
+ document.sidecar.image.attached?
189
+
190
+ # Can the image size be manipulated?
191
+ document.sidecar.image.variable?
192
+
193
+ # Example image_tag with resize
194
+ <%= image_tag document.sidecar.image.variant(resize: "100x100"), {class: 'media-object'} %>
195
+
196
+ ```
197
+
198
+ You'll definitely want to update this file to fit your own design's needs.
199
+
91
200
  ## Development
92
201
 
93
202
  ```bash
203
+
204
+ # Run test suite
94
205
  bundle exec rake ci
206
+
207
+ # Launch test app server
95
208
  cd .internal_test_app/
96
209
  rake geoblacklight:server
210
+
211
+ # Load test fixtures
212
+ bundle exec rake gblsci:sample_data:ingest['<FULL_LOCAL_PATH_TO>/geoblacklight_sidecar_images/spec/fixtures/files']
213
+
214
+ # Run harvest
215
+ bundle exec rake gblsci:images:harvest_all
216
+
217
+ # Tail log
218
+ tail -f log/image_service_development.log
219
+
97
220
  ```
98
221
 
99
- Now you'll have an instance of GBLSI running. Follow the rake tasks above to ingest some data and harvest thumbnails.
222
+ [See Localhost Results](http://localhost:3000/?per_page=50&q=&search_field=all_fields)
100
223
 
101
224
  ## TODOs
102
225
 
103
226
  * ~~0.0.1 - Initial gem~~
104
227
  * ~~0.1.0 - Prioritize local thumbnail solr field~~
105
- * 0.2.0 - Forgo attaching placeholder imagery
106
- * 0.3.0 - Add Statesman (state machine library)
107
- * 1.0.0 - Rails 5.2 branch / Switch to ActionStorage
228
+ * ~~0.2.0 - Forgo attaching placeholder imagery~~
229
+ * ~~0.3.0 - Add Statesman (state machine library)~~
230
+ * ~~0.4.0 - Rails 5.2 branch / Switch to ActionStorage~~
231
+ * 0.5.0 to 0.9.0 - Feedback; Improve test coverage; Collect additional real-world issues
232
+ * 1.0.0 - Final 5.2 release
@@ -1,10 +1,12 @@
1
- class CreateSolrDocumentSidecars < ActiveRecord::Migration[5.1]
1
+ class CreateSolrDocumentSidecars < ActiveRecord::Migration[5.2]
2
2
  def change
3
3
  create_table :solr_document_sidecars do |t|
4
4
  t.string "document_id"
5
5
  t.string "document_type"
6
6
  t.string "image"
7
- t.index ["document_type", "document_id"], name: "solr_document_sidecars_solr_document"
7
+ t.integer "version", :limit => 8
8
+
9
+ t.index ["document_type", "document_id"], name: "sidecars_solr_document"
8
10
 
9
11
  t.timestamps
10
12
  end
@@ -0,0 +1,29 @@
1
+ class CreateSidecarImageTransitions < ActiveRecord::Migration[5.2]
2
+ def change
3
+ create_table :sidecar_image_transitions do |t|
4
+ t.string :to_state, null: false
5
+ t.text :metadata
6
+ t.integer :sort_key, null: false
7
+ t.bigint :solr_document_sidecar_id, null: false
8
+ t.boolean :most_recent
9
+
10
+ # If you decide not to include an updated timestamp column in your transition
11
+ # table, you'll need to configure the `updated_timestamp_column` setting in your
12
+ # migration class.
13
+ t.timestamps null: false
14
+ end
15
+
16
+ # Foreign keys are optional, but highly recommended
17
+ add_foreign_key :sidecar_image_transitions, :solr_document_sidecars
18
+
19
+ add_index(:sidecar_image_transitions,
20
+ [:solr_document_sidecar_id, :sort_key],
21
+ unique: true,
22
+ name: "index_sidecar_image_transitions_parent_sort")
23
+ add_index(:sidecar_image_transitions,
24
+ [:solr_document_sidecar_id, :most_recent],
25
+ unique: true,
26
+
27
+ name: "index_sidecar_image_transitions_parent_most_recent")
28
+ end
29
+ end
@@ -18,10 +18,12 @@ Gem::Specification.new do |s|
18
18
  s.executables = s.files.grep(%r{^bin/}) { |f| File.basename(f) }
19
19
  s.require_paths = ['lib']
20
20
 
21
- s.add_dependency 'carrierwave', '~> 1.2'
22
- s.add_dependency 'geoblacklight', '~> 1.7'
21
+ s.add_dependency 'geoblacklight', '~> 1.9'
23
22
  s.add_dependency 'mini_magick', '~> 4.8'
24
- s.add_dependency 'rails', '>= 4.2', '< 6'
23
+ s.add_dependency 'image_processing', '~> 1.6'
24
+ s.add_dependency 'statesman', '~> 3.4'
25
+ s.add_dependency 'mimemagic', '~> 0.3'
26
+ s.add_dependency 'rails', '>= 5.2', '< 6'
25
27
 
26
28
  s.add_development_dependency 'byebug'
27
29
  s.add_development_dependency 'capybara'
@@ -3,17 +3,16 @@
3
3
  require 'rails/generators'
4
4
 
5
5
  module GeoblacklightSidecarImages
6
- class UploadersGenerator < Rails::Generators::Base
6
+ class ConfigGenerator < Rails::Generators::Base
7
7
  source_root File.expand_path('templates', __dir__)
8
8
 
9
9
  desc <<-DESCRIPTION
10
10
  This generator makes the following changes to your application:
11
- 1. Creates an app/uploaders directory
12
- 2. Creates uploader models within the app/uploaders directory
11
+ 1. Copies config files to host config
13
12
  DESCRIPTION
14
13
 
15
- def create_image_uploader
16
- directory 'uploaders', 'app/uploaders'
14
+ def create_store_image_jobs
15
+ copy_file 'config/initializers/statesman.rb', 'config/initializers/statesman.rb'
17
16
  end
18
17
  end
19
18
  end
@@ -16,16 +16,6 @@ GBLSI_THUMBNAIL_FIELD: 'thumbnail_path_ss'"
16
16
  end
17
17
  end
18
18
 
19
- def add_carrierwave_require
20
- inject_into_file 'config/application.rb', after: "require 'rails/all'" do
21
- "\n require 'carrierwave'"
22
- end
23
- end
24
-
25
- def generate_geoblacklight_assets
26
- generate 'geoblacklight_sidecar_images:assets'
27
- end
28
-
29
19
  def generate_geoblacklight_example_docs
30
20
  generate 'geoblacklight_sidecar_images:example_docs'
31
21
  end
@@ -42,14 +32,18 @@ GBLSI_THUMBNAIL_FIELD: 'thumbnail_path_ss'"
42
32
  generate 'geoblacklight_sidecar_images:services'
43
33
  end
44
34
 
45
- def generate_geoblacklight_uploaders
46
- generate 'geoblacklight_sidecar_images:uploaders'
47
- end
48
-
49
35
  def generate_geoblacklight_views
50
36
  generate 'geoblacklight_sidecar_images:views'
51
37
  end
52
38
 
39
+ def generate_action_storage
40
+ rake 'active_storage:install'
41
+ end
42
+
43
+ def generate_geoblacklight_config
44
+ generate 'geoblacklight_sidecar_images:config'
45
+ end
46
+
53
47
  def bundle_install
54
48
  Bundler.with_clean_env do
55
49
  run 'bundle install'
@@ -24,10 +24,18 @@ module GeoblacklightSidecarImages
24
24
  def include_sidecar_solrdocument
25
25
  sidecar = <<-"SIDECAR"
26
26
  def sidecar
27
- SolrDocumentSidecar.find_or_create_by!(
27
+ # Find or create, and set version
28
+ sidecar = SolrDocumentSidecar.where(
28
29
  document_id: id,
29
30
  document_type: self.class.to_s
30
- )
31
+ ).first_or_create do |sc|
32
+ sc.version = self._source["_version_"]
33
+ end
34
+
35
+ sidecar.version = self._source["_version_"]
36
+ sidecar.save
37
+
38
+ sidecar
31
39
  end
32
40
  SIDECAR
33
41
 
@@ -41,6 +49,20 @@ module GeoblacklightSidecarImages
41
49
  )
42
50
  end
43
51
 
52
+ def create_sidecar_image_transition
53
+ copy_file(
54
+ 'models/sidecar_image_transition.rb',
55
+ 'app/models/sidecar_image_transition.rb'
56
+ )
57
+ end
58
+
59
+ def create_sidecar_image_state_machine
60
+ copy_file(
61
+ 'models/sidecar_image_state_machine.rb',
62
+ 'app/models/sidecar_image_state_machine.rb'
63
+ )
64
+ end
65
+
44
66
  def create_wms_rewrite_concern
45
67
  copy_file(
46
68
  'models/concerns/wms_rewrite_concern.rb',
@@ -0,0 +1,5 @@
1
+ require 'statesman'
2
+
3
+ Statesman.configure do
4
+ storage_adapter(Statesman::Adapters::ActiveRecord)
5
+ end
@@ -3,8 +3,15 @@
3
3
  class StoreImageJob < ApplicationJob
4
4
  queue_as :default
5
5
 
6
- def perform(document_hash)
7
- doc = SolrDocument.new(document_hash)
8
- ImageService.new(doc).store
6
+ def perform(solr_document_id)
7
+ cat = CatalogController.new
8
+ response, document = cat.fetch(solr_document_id)
9
+
10
+ metadata = Hash.new
11
+ metadata['solr_doc_id'] = document.id
12
+ metadata['solr_version'] = document.sidecar.version
13
+
14
+ document.sidecar.image_state.transition_to!(:queued, metadata)
15
+ ImageService.new(document).store
9
16
  end
10
17
  end
@@ -0,0 +1,19 @@
1
+ class SidecarImageStateMachine
2
+ include Statesman::Machine
3
+
4
+ state :initialized, initial: true
5
+ state :queued
6
+ state :processing
7
+ state :succeeded
8
+ state :failed
9
+ state :placeheld
10
+
11
+ # Queued => Background Job Init
12
+ # Processing => Failed, Placeheld, Succeeded
13
+ transition from: :initialized, to: [:queued, :processing]
14
+ transition from: :queued, to: [:queued, :processing]
15
+ transition from: :processing, to: [:queued, :processing, :placeheld, :succeeded, :failed]
16
+ transition from: :placeheld, to: [:queued, :processing]
17
+ transition from: :failed, to: [:queued, :processing]
18
+ transition from: :succeeded, to: [:queued, :processing]
19
+ end