kithe 2.1.0 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e9a96dd80fc482ea07bd35456a0cf8445b722354cd6fe1fda10460398a0d7846
-  data.tar.gz: 0ccb6788a1ac0f88a47202767f1a5acb7dc12031cc4293ec27e09ded198b8daf
+  metadata.gz: df697bf6cd0128be5ee217fb3eb191af55994b1284e1e05f75da73667e92d8f3
+  data.tar.gz: f388992232e60a99f7a6f7a2743e47a1690152665b9c85adbdf34457ce822104
 SHA512:
-  metadata.gz: 46bcc8b60dd795e4f6e063aef6c7702184d853f2266f3fb5db4636074379faf3a45a1baae83e0d0358cea6fa8c8620a43727d9ee90cffb511c28e371072fe8ad
-  data.tar.gz: d4250716ef0b4c32c2b993a1989290a59afdb56be2159a8f814bd3b62bfe5e853845188b3200bc1c9be2f8c8f5bad950276e1286324d9f28d890e50c2ccf7a86
+  metadata.gz: '079a440ca120cc56e4a624b335cab990b2dc0f8a7df73821f251ecae4bf17f91000e9f57bc59c01d6a378577c871d1bc5d8a31ccee3a49a8402459edb532c7c0'
+  data.tar.gz: 73ba63882d5df383011c354c648b82f422891018679c8397a13143a937f9f9a12142eac18ea22d3162bb0f3f127b7ba9234a7309a18bd6329013c598ad17737d

data/README.md

@@ -7,54 +7,32 @@ An experiment in shareable tools/components for building a digital collections a
 
 Kithe is a toolkit for building digital collections/repository applications in Rails. It comes out of experience in the [samvera](https://samvera.org/) community of open source library-archives-museums digital collections/preservation work (but is not a samvera project).
 
-Kithe does not use fedora or valkyrie, but stores all metadata using ActiveRecord.  Kithe requires you use postgres 9.5+ as your db. While kithe provides some additional architecture and support on top of "ordinary" ActiveRecord, it tries to mostly let you develop as you would any Rails app, inclduing all choices and you'd normally have, and taking advantage of standard ActiveRecord-based functionality.  Kithe bases it's file handling framework on [shrine](https://shrinerb.com), supporting cloud or local file storage.
+Kithe does not use fedora or valkyrie, but stores all metadata using ActiveRecord.  Kithe requires you use postgres 9.5+ as your db. It uses [shrine](https://shrinerb.com) for file-handling/asset-storing and tries to support developing your app as a normal Rails/ActiveRecord app. It will not give you a working turnkey application, but is a collection of tools for building an app with certain patterns.
 
-Kithe will not give you a working "turnkey" application. It is a collection of tools to help you write a Rails app. You may end up using some but not all of kithe's tools.  The range of tools provided and areas of an app given some support in kithe will probably grow over time, in hopefully a careful and cautious way.
+Kithe provides tools to supports these architectural patterns:
 
-In that it provides tools and not a turnkey app, develping an app based on kythe in some ways similar to developing an app based on [valkyrie](https://github.com/samvera-labs/valkyrie). They both provide basic architecture for modelling/persistence, although in quite different ways. Kithe also provides tools in addition to modelling/persistence, but does _not_ provide the data-mapper/repository pattern valkyrie does, or any built-in abstraction for persisting anywhere but a postgres DB.
+* [Modelling and Persistence](./guides/modelling.md):
+  * A Collection/Work/Asset model based on Samvera/PCDM, using rails Single-Table Inheritance to support hetereogenous associations with efficient rdbms lookup.
+  * Using Postgres JSONB for "schema-less" flexible storage, via [attr_json](https://github.com/jrochkind/attr_json), supporting complex structured nested repeatable data values.
+  * [Work representatives](./guides/work_representative.md) via ActiveRecord association, using postgres recursive CTE's to compute the "leaf" representative, designed to support efficient use of the DB including pre-loading leaf representatives.
+  * UUIDv4's as internal primary keys, but also provide a "friendlier_id" with a shorter unique alphanumeric identifier for URLs and other UI. By default they are supplied by a postgres stored procedure, but your code can set them to whatever you like.
 
-If you are comparing it to a "solution bundle" digital collections platform, kithe may seem like more work. But experience has shown me that in our domain, historically "solution bundles" can be less of a "turnkey" approach than they seem, and can have greater development cost over total app lifecycle than anticipated. If you have similar experience that leads you to consider a more 'bespoke' app approach -- you may want to consider kithe. We hope to provide architecturally simple support and standardization for your custom app, taking care of some of the common "hard parts" and leaving you with flexibility to build out the app that meets your needs.
-
-Kithe has beeen developed in tandem with the Science History Institute's in-development [replacement digital collections](https://github.com/sciencehistory/scihist_digicoll) app, and you can look there for a model/canonical/demo kithe use.
-
-The Science History Institute app is live and working, so kithe is 1.0. We are serious about [semantic verisioning](https://semver.org/) and will endeavor to release backwards breaking changes only with a major release, and minimize major releases.
-
-While it is working well for us, since it hasn't had wide use, it could still be considered somewhat of an experiment. But you are invited to try it out and see how it works. You are welcome to use it, but also welcome to copy any code or just ideas from kithe.
-
-Any questions or feedback of any kind are very welcome and encouraged, in the github project issues, samvera slack, or wherever is convenient.
-
-
-# Kithe parts
-
-Some guide documentation is available to explain each of kithe's major functionality areas. Definitely start with the modelling guide.
-
-* [Modelling and Persistence](./guides/modelling.md):  It can be somewhwat challenging to figure out a good way to model our data in an rdbms. We give you a hopefully flexible and understandable architecture that is designed to support efficient performance. It's influenced by PCDM and traditional samvera modelling. It's based on [attr_json](https://github.com/jrochkind/attr_json) to let you model arbitrary and complex object-oriented data that gets persisted as a serialized json hash. It uses rails Single Table Inheritance to support hetereogenous associations and collections. The modelling classes in some places use postgres-specific features for efficiency.
-
-  * [Work representatives](./guides/work_representative.md). Built in associations to support "representative", using postgres recursive CTE's to compute the "leaf" representative, designed to support efficient use of the DB including pre-loading leaf representatives.
-
-  * Kithe objects use UUIDv4's as internal primary keys, but also provide a "friendlier_id" with a shorter unique alphanumeric identifier for URLs and other UI. By default they are supplied by a postgres stored procedure, but your code can set them to whatever you like.
-
-* [Form support](./guides/forms.md): Dealing with complex and _repeatable_ data, as our modelling layer allows, can be tricky in an HTML form. We supply javascript-backed Rails form input help for repeatable and compound/nested data.
+* [Form support](./guides/forms.md):  Easy Rails-like forms for that complex nested and repeatable form data, leaning on simple_form.
   * An extension to Rails "strong parameters" that make some common patterns for
-    embedded JSON attributes more convenient, [Kithe::Parameters](./app/models/kithe/parameters.rb).
-
-* [File handling](./guides/file_handling.md): Handling files is at the core of digital repository use cases. We need a file handling framework that is flexible, predictable and reliable, and architected for performance. We try to give you one based on the [shrine](https://shrinerb.com) file attachment toolkit for ruby.
+    embedded JSON attributes more convenient, [Kithe::Parameters](./app/models/kithe/parameters.rb)
 
-  * [Derivatives](./guides/derivatives.md) A flexible and reliable derivatives architecture, designed to ensure data consistency without race conditions, and support efficient DB usage patterns.
+* [File handling](./guides/file_handling.md): A framework that let's you easily plug in your own custom characterization and derivatives handling, to be handled in an efficient and flexible way, ordinarily using background jobs. Implemented on top of [shrine](https://shrinerb.com).
+  * [Derivatives](./guides/derivatives.md) handling ensures data consistency without race conditions, and efficient querying patterns, letting you plugin custom derivatives creation, with some standard routines included.
 
-* [Solr Indexing](./guides/solr_indexing.md): Uses [traject](https://github.com/traject/traject) for defining mappings from your model objects to what you want in a Solr index. Uses ActiveRecord callbacks to automatically sync saves to solr, with many opportunities for customization.
+* [Solr Indexing](./guides/solr_indexing.md): Built-in Solr indexing using [traject](https://github.com/traject/traject) for defining mappings from your model objects to what you want in a Solr index. Uses ActiveRecord callbacks to automatically sync saves to solr, with many opportunities for customization.
   * Not coupled to any other kithe components, could be used independently, hypothetically on any ActiveRecord model.
-  * Written after review of "prior art"  in [sunspot](https://github.com/sunspot/sunspot) and [searchkick](https://github.com/ankane/searchkick) (which both used AR callback-based indexing), and others.
-
-* A [recommended approach for using Blacklight](./guides/blacklight_approach.md) with search result view templates based on actual ActiveRecord models. It is totally optional to use Blacklight at all with kithe, or to use this approach if you do.
 
-### Also
+* A [recommended approach for using Blacklight](./guides/blacklight_approach.md) with search result view templates based on actual ActiveRecord models. Blacklight use is optional with kithe, but kithe works well with blacklight.
 
-* [Kithe::Parameters](./app/models/kithe/parameters.rb) provides some shortcuts around Rails "strong params" for attr_json serialized attributes.
+* Assorted optional utilities
+  * [Kithe::ConfigBase](./app/models/kithe/config_base.rb) A totally optional solution for managing environmental config variables.
 
-* [Kithe::ConfigBase](./app/models/kithe/config_base.rb) A totally optional solution for managing environmental config variables.
-
-* [ArrayInclusionValdaitor](./app/validators/array_inclusion_validator.rb) Useful for validating on attr_json arrays of primitives.
+  * [ArrayInclusionValdaitor](./app/validators/array_inclusion_validator.rb) Useful for validating on attr_json arrays of primitives.
 
 ## Setting up your app to use kithe
 
@@ -70,11 +48,28 @@ So you want to start an app that uses kithe. We should later provide better 'get
 
 * Specific additional pre-requisites/requirements can sometimes be found in individual feature docs. And include the Javascript from [cocoon](https://github.com/nathanvda/cocoon), for form support for repeatable-field editing forms. We haven't quite figured out our preferred sane approach for sharing Javascript via kithe.
 
-Note that at present kithe will end up forcing your app to use `:sql` [style schema dumps](https://guides.rubyonrails.org/v3.2.8/migrations.html#types-of-schema-dumps). We may try to fix this.
+
+## Why kithe?
+
+Kithe tries to let you develop your app like "an ordinary Rails app" (in all it's possible variations), while handling some of the rough spots common to the kinds of modelling and administration common to digital collections domains.  But developers should be able to use standard Rails patterns and skills to develop an app to your specific local needs, familiar, no more complicated than building any other Rails app. You add features to a kithe app just like building Rails, using whatever patterns you like. We support modern Rails versions, 5.2+.
+
+In that kithe provides tools and not a turnkey app, develping an app based on kythe in some ways similar to developing an app based on [valkyrie](https://github.com/samvera-labs/valkyrie) (but not hyrax). They both provide basic architecture for modelling/persistence, although in quite different ways. Kithe also provides tools in addition to modelling/persistence, but does _not_ provide the data-mapper/repository pattern valkyrie does, or any built-in abstraction for persisting anywhere but a postgres DB.
+
+If you are comparing it to a "solution bundle" digital collections platform like hyrax, kithe may seem like more work. But experience has shown us that in our domain, "solution bundles" can turn out less of a "turnkey" approach than they seem, and can have greater development cost over total app lifecycle than anticipated. If you have similar experience that leads you to consider a more 'bespoke' app approach -- you may want to consider kithe. We hope to provide architecturally simple support and standardization for your custom app, taking care of some of the common "hard parts" and leaving you with flexibility to build out the app that meets your needs.
+
+Kithe has beeen developed in tandem with the Science History Institute's in-development [replacement digital collections](https://github.com/sciencehistory/scihist_digicoll) app, which has been in production for several years using kithe.
+
+[The University of Minnesota found kithe](https://docs.google.com/presentation/d/1Z4AoIDOaxbY4pt3mDhNt6MfUs6VIMjysKKmaYQpjuk8/edit?usp=sharing) to pair well with GeoBlacklight, an easy way to provide the persistence layer and metadata editing UI that blacklight on it's own lacks.
+
+We are serious about [semantic verisioning](https://semver.org/) and will endeavor to release backwards breaking changes only with a major release, and minimize major releases.
+
+Kithe is working well for us, but has had limited (but non-zero) adoption from other institutions. It's still somewhat of an experiment, but one we think is going well. If you would consider developing a digital collections/repository app in "just Rails", we think it's worth investigating if kithe can save you some trouble in some rough common use cases. You are invited to try it out and see how it works, using kithe directly, or copying any code or just ideas from kithe.
+
+Any questions or feedback of any kind are very welcome and encouraged!  In the github project issues, samvera slack, or wherever is convenient.
 
 ## To be done
 
-Considering some blacklight integration support.
+Considering some additional blacklight integration support, is any needed?
 
 Other components/features may become more clear as we continue to develop. It's possible that kithe won't (at least for a long time) contain controllers themselves (it may contain some helper methods for controllers), or generalized permissions architecture. Both of these are some of the things most particular to specific apps, that are hard to generalize without creating monsters.
 
@@ -85,8 +80,6 @@ This is a Rails 'engine' whose template was created with: `rails plugin new kith
 
 * Note we have chosen not to make it 'mountable' or 'isolated', I think that would be inappropriate for this kind of gem. It _is_ an engine so it can hook into Rails load paths and config as needed.
 
-
-
 * Note we are currently using the standard rails-generated dummy app in spec/dummy for testing, rather than [engine_cart](https://github.com/cbeer/engine_cart) or [combustion](https://github.com/pat/combustion).
   * Before you run the tests for the first time, create the database by running: `rails db:setup`. This will create two databases, kithe_development and kithe_test.
   * Some of the rspec tests depend on [FFmpeg](https://ffmpeg.org/) for testing file derivative transformations. Mac users can install [ffmpeg via homebrew](https://formulae.brew.sh/formula/ffmpeg): `brew install ffmpeg`

data/app/indexing/kithe/indexable/thread_settings.rb

@@ -97,6 +97,8 @@ module Kithe
         # only call on-finish if we have a writer, batch writers are lazily
         # created and maybe we never created one
         if @writer
+          # if we created the writer ourselves locally and nobody
+          # specified an on_finish, close our locally-created writer.
           on_finish = if @local_writer && @on_finish.nil?
             proc {|writer| writer.close }
           else
@@ -105,7 +107,7 @@ module Kithe
           on_finish.call(@writer) if on_finish
         end
 
-        Thread.current[THREAD_CURRENT_KEY] = @original_thread_current_settings
+        Thread.current[THREAD_CURRENT_KEY] = @original_settings
       end
 
       private

data/app/indexing/kithe/indexable.rb

@@ -120,6 +120,9 @@ module Kithe
     #
     # By default will use a per-update writer, or thread/block-specific writer configured with `self.index_with`,
     # or you can pass one in.
+    #
+    # This method is part of Kithe API, including allowing local apps to override! Backwards
+    # compatibilty matters for semver with any change to method signature.
     def update_index(mapper: kithe_indexable_mapper, writer:nil)
       RecordIndexUpdater.new(self, mapper: mapper, writer: writer).update_index
     end

data/app/models/kithe/asset.rb

@@ -81,31 +81,11 @@ class Kithe::Asset < Kithe::Model
     source = file
     return false unless source
 
-    #local_files = file_attacher.process_derivatives(:kithe_derivatives, only: only, except: except, lazy: lazy)
-    local_files = _process_kithe_derivatives_without_download(source, only: only, except: except, lazy: lazy)
+    local_files = file_attacher.process_derivatives(:kithe_derivatives, only: only, except: except, lazy: lazy)
 
     file_attacher.add_persisted_derivatives(local_files)
   end
 
-  # Working around Shrine's insistence on pre-downloading original before calling derivative processor.
-  # We want to avoid that, so when our `lazy` argument is in use, original does not get eagerly downloaded,
-  # but only gets downloaded if needed to make derivatives.
-  #
-  # This is a somewhat hacky way to do that, loking at the internals of shrine `process_derivatives`,
-  # and pulling them out to skip the parts we don't want. We also lose shrine instrumentation
-  # around this action.
-  #
-  # See: https://github.com/shrinerb/shrine/issues/470
-  #
-  # If that were resolved, the 'ordinary' shrine thing would be to replace calls
-  # to this local private method with:
-  #
-  #     file_attacher.process_derivatives(:kithe_derivatives, only: only, except: except, lazy: lazy)
-  #
-  private def _process_kithe_derivatives_without_download(source, **options)
-    processor = file_attacher.class.derivatives_processor(:kithe_derivatives)
-    local_files = file_attacher.instance_exec(source, **options, &processor)
-  end
 
   # Just a convennience for file_attacher.add_persisted_derivatives (from :kithe_derivatives),
   # feel free to use that if you want to add more than one etc.  By default stores to
@@ -144,6 +124,14 @@ class Kithe::Asset < Kithe::Model
     result && result.values.first
   end
 
+  # Like #update_derivative, but can update multiple at once.
+  #
+  #     asset.update_derivatives({ "big_thumb" => big_thumb_io, "small_thumb" => small_thumb_io })
+  #
+  # Options from kithe `add_persisted_derivatives`/shrine `add_derivative` supported.
+  #
+  #     asset.update_derivatives({ "big_thumb" => big_thumb_io, "small_thumb" => small_thumb_io }, delete_false)
+  #
   def update_derivatives(deriv_hash, **options)
     file_attacher.add_persisted_derivatives(deriv_hash, **options)
   end

data/app/validators/array_inclusion_validator.rb

@@ -42,7 +42,7 @@ class ArrayInclusionValidator < ActiveModel::EachValidator
 
     unless not_allowed_values.blank?
       formatted_rejected = not_allowed_values.uniq.collect(&:inspect).join(",")
-      record.errors.add(attribute, :inclusion, options.except(:in).merge!(rejected_values: formatted_rejected, value: value))
+      record.errors.add(attribute, :inclusion, **options.except(:in).merge!(rejected_values: formatted_rejected, value: value))
     end
   end
 end

data/lib/kithe/version.rb

@@ -1,6 +1,3 @@
 module Kithe
-  # not sure why rubygems turned our alphas into 2.0.0.pre.alpha1, inserting
-  # "pre". We need to do same thing with betas to get version orderings
-  # appropriate.
-  VERSION = '2.1.0'
+  VERSION = '2.2.0'
 end

data/lib/shrine/plugins/kithe_derivative_definitions.rb

@@ -10,7 +10,11 @@ class Shrine
 
         # Register our derivative processor, that will create our registered derivatives,
         # with our custom options.
-        uploader::Attacher.derivatives(:kithe_derivatives) do |original, **options|
+        #
+        # We do download: false, so when our `lazy` argument is in use, original does not get eagerly downloaded,
+        # but only gets downloaded if needed to make derivatives. This is great for performance, especially
+        # when running batch job to add just missing derivatives.
+        uploader::Attacher.derivatives(:kithe_derivatives, download: false) do |original, **options|
           Kithe::Asset::DerivativeCreator.new(self.class.kithe_derivative_definitions,
             source_io: original,
             shrine_attacher: self,
@@ -44,10 +48,12 @@ class Shrine
         # Tempfile and Dir.mktmpdir may be useful.
         #
         # If in order to do your transformation you need additional information about the original,
-        # just add a `record:` keyword argument to your block, and the Asset object will be passed in:
+        # just add a `attacher:` keyword argument to your block, and a `Shrine::Attacher` subclass
+        # will be passed in. You can then get the model object from `attacher.record`, or the
+        # original file as a `Shrine::UploadedFile` object with `attacher.file`.
         #
-        #     define_derivative :thumbnail do |original_file, record:|
-        #        record.width, record.height, record.content_type # etc
+        #     define_derivative :thumbnail do |original_file, attacher:|
+        #        attacher.record.title, attacher.file.width, attacher.file.content_type # etc
         #     end
         #
         # Derivatives are normally uploaded to the Shrine storage labeled :kithe_derivatives,

data/lib/shrine/plugins/kithe_persisted_derivatives.rb

@@ -26,6 +26,11 @@ class Shrine
         # Like the shrine `add_derivatives` method, but also *persists* the
         # derivatives (saves to db), in a realiably concurrency-safe way.
         #
+        # For ruby 3 compatibility, make sure you supply local_files as a hash
+        # literal with curly braces:
+        #
+        #     attacher.add_persisted_derivatives({ derivative_name1: io_obj1, deriv2: io2 })
+        #
         # Generally can take any options that shrine `add_derivatives`
         # can take, including custom `storage` or `metadata` arguments.
         #

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kithe
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 2.2.0
 platform: ruby
 authors:
 - Jonathan Rochkind
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-04-13 00:00:00.000000000 Z
+date: 2021-11-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -70,14 +70,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.2'
+        version: '3.3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.2'
+        version: '3.3'
 - !ruby/object:Gem::Dependency
   name: shrine-url
   requirement: !ruby/object:Gem::Requirement
@@ -410,7 +410,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
 summary: Shareable tools/components for building a digital collections app in Rails.