digital_scriptorium 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 62b823f25e2940c6a68ee4ea8db949cebbdc95f9d31682d72a60dbbb62c0a0ff
-  data.tar.gz: 53c69f7e20af8b7efc327214c4bfbb8378f6d80133106458231277961f4ed613
+  metadata.gz: c72c2b9f90471cda9645a21bfdbec8e617d92372c7139eaab149ace86ae1d2ba
+  data.tar.gz: 2b594eec7517dc48419e94c33571835108d1abc16c5fb2a4348c524efb083dd7
 SHA512:
-  metadata.gz: 20ae390598e32c3276426c98dd437cd9d097e485d50bcb86757058defbbce42408a3a00b8c20fec9f86f2010078483bec4e1cd3053fc39ad42a462cf283e30e4
-  data.tar.gz: e200d3a053fba432f9819c68ceb3c48db5ad7d6daafc648f39ef5039b0fba345275c04f00215aff8c369fae2cf897e3eef31152bf93d375fdfa560b55b86564c
+  metadata.gz: 0c26cb8f29768b8c8f49a44774021f18aa4e0c0fa11ae98672ea3431eccac29d23eaae5a194695e238e47edd46387ffdd8ca8a32a9e49a3a8bb1aabb489de508
+  data.tar.gz: 25e1557a0020df6f64fd4b6fe45f48ee0bbf8521810e8d719e7151045913a3d77dcd3e37cac30a0180fd972a7932085dac3cb2b819412fa838ad75043776e7f3

data/doc/overview.md

@@ -2,12 +2,12 @@
 
 For a general description of the Wikibase data model, see [Wikibase/DataModel](https://www.mediawiki.org/wiki/Wikibase/DataModel) on mediawiki.org.
 
-The Digital Scriptorium Wikibase data export is a JSON-formatted array of Wikibase entities. The bulk of the entities in the export consist of triplets that together form a meta-record consisting of one each of the DS Catalog core model types: manuscipts, holdings, and records. The export also contains entities representing property definitions and authoritative references to common topics.
+The Digital Scriptorium Wikibase data export is a JSON-formatted array of Wikibase entities. The bulk of the entities in the export consist of the DS Catalog core model types: manuscipts (Q1), holdings (Q2), and DS 2.0 records (Q3). The export also contains entities representing property definitions and authoritative references to common topics.
 
 The [ExportRepresenter](../lib/digital_scriptorium/export_representer.rb) class can be used to deserialize an export in its entirety. The resulting [Export](../lib/digital_scriptorium/export.rb) object is essentially an array of Item and Property objects. Entities in the export are modeled using domain-specific classes provided by the [wikibase_representable](https://rubygems.org/gems/wikibase_representable) gem, such as Items, Properties, Statements (also known as Claims), and Snaks, which represent the primary claim of any statement as well as any qualifiers. Convenience methods are also provided to facilitate extracting data values.
 
-The conversion script [wikibase_to_solr_new.rb](../wikibase_to_solr_new.rb) proceeds by deserializing the export and converting the resulting array of Wikibase objects to a hash keyed by entity ID. It then iterates over the elements of the hash. When it finds a record item based on the value of its instance-of (P16) claim, it retrieves the linked manuscript item, as well as the holding item linked in turn to the manuscript item, from the export hash by entity ID. It then iterates over the claims attached to manuscript, holding, and record in turn, extracting the Solr fields requested based on the property ID that is the subject of the claim and adding them to the Solr record to be produced for the meta-record. Claims for most properties are transformed to Solr fields using a generic algorithm implemented in [ClaimTransformer](../lib/digital_scriptorium/claim_transformer.rb). Name and date claims require some special handling, and are handled in dedicated claim transformer classes ([NameClaimTransformer](../lib/digital_scriptorium/name_claim_transformer.rb) and [DateClaimTransformer](../lib/digital_scriptorium/date_claim_transformer.rb) respectively). After all claims from the manuscript, holding, and record have been processed, the resulting Solr record is written to the output file.
+The conversion script [wikibase_to_solr.rb](https://github.com/mdholloway/hxs-blacklight/blob/main/lib/wikibase_to_solr.rb) proceeds by deserializing the export and converting the resulting array of Wikibase objects to a hash keyed by entity ID. It then iterates over the elements of the hash. When it finds a DS 2.0 record item based on the value of its instance-of (where P16 is Q3) claim, it retrieves the linked manuscript item (P1, described manuscript) from the export hash by entity ID. From the manuscript, in turn, it retrieves the ID of the item containing current holding information (P2, manuscript holding), and retrieves that too from the export hash. With the manuscript, current holding, and record items obtained, it iterates over each, extracting the Solr fields requested based on the property ID that is the subject of the claim and adding them to the Solr record to be produced. After all claims from the manuscript, holding, and record have been processed, the resulting Solr record is written to the output file. The script is written so as not to rely on the structure of the export file beyond that it will be a JSON array consisting of all entities in the DS 2.0 Wikibase, with record items linked to manuscript items and manuscript items linked to holding items by P3 (described manuscript) and P2 (holding) claims respectively.
 
-The specific Solr fields produced for each claim are controlled by the configuration file [property_config.yml](../property_config.yml). This file also defines the prefix (representing the property name) to be attached to each field for a given property, and whether a claim based on the property might have a related authority qualifier. 
+Solr field extraction logic is encapsulated in the Transformer classes. The [BaseClaimTransformer](../lib/digital_scriptorium/transformers/base_claim_transformer.rb) class sets out the basic contract, which consists of three methods: `display_values`, `search_values`, and `facet_values`. These methods return the collections of values to be included in the `_display`, `_search`, and `_facet` fields for the claim in the Solr object. For a title (P10) claim, for example, they would return the values to be used in the `title_display`, `title_search`, and `title_facet` fields. The remaining Transformer classes build on BaseClaimTransformer in various ways. For some claim types, the Transformer simply extracts the recorded value and returns it in one or more of the `_values` methods. For other claim types, it is expected that a claim will be qualified with a representation of the recorded value in its original script, or with references to a standard title or value from an authority file. This logic is contained in the [QualifiedClaimTransformer](../lib/digital_scriptorium/transformers/qualified_claim_transformer.rb) class. For these claim types, the standard title or value from authority file is returned in the `facet_values` collection. For claim types where the recorded value should be provided as a facet value in the absence of a qualifier, the [QualifiedClaimTransformerWithFacetFallback](../lib/digital_scriptorium/transformers/qualified_claim_transformer_with_facet_fallback.rb) class is provided. Finally, the [LinkClaimTransformer](../lib/digital_scriptorium/transformers/link_claim_transformer.rb) class handles a couple of claim types for which the value to be extracted is a URL. The [Transformers](../lib/digital_scriptorium/transformers.rb) class contains the mapping of claim property IDs to Transformer classes, as well as the prefixes to be used in the Solr fields based on the property name, and provides factory methods used by the conversion script to obtain Transformers as it iterates over claims.
 
-The script was written so as not to rely on the structure of the export file beyond that it will be a JSON array consisting of all entities in the DS 2.0 Wikibase, with record items linked to manuscript items and manuscript items linked to holding items by P3 (described manuscript) and P2 (holding) claims respectively.
+The `_search` and `_facet` fields contain values pulled directly from the source Wikibase data. The `_display` field values contain serialized JSON objects that are used to support linked data bars beneath recorded values when viewing item details in the Catalog. These objects contain a `recorded_value` property with the recorded value and an optional `original_script` property with the value in original script where present. Additionally, where the recorded value is qualified by one or more qualifiers, the object will contain a `linked_terms` array to support sets of one or more linked data bars beneath the recorded value. The objects in this array will contain a `label` property that can be passed to a faceted search, as well as an optional `source_url` property that contains a link to an external vocabulary item.

data/lib/digital_scriptorium/ds_item.rb

@@ -3,46 +3,12 @@
 require 'wikibase_representable'
 
 module DigitalScriptorium
-  # Represents a Digital Scriptorium item
+  # Represents a generic Digital Scriptorium item
   class DsItem < WikibaseRepresentable::Model::Item
-    def instance_of_claims
-      claims_by_property_id PropertyId::INSTANCE_OF # P16
-    end
-
-    def ds_id
-      claims_by_property_id(PropertyId::DS_ID)&.first&.data_value # P1
-    end
-
-    def holding_ids
-      claims_by_property_id(PropertyId::MANUSCRIPT_HOLDING)&.map(&:entity_id_value) # P2
-    end
-
-    def described_manuscript_id
-      claims_by_property_id(PropertyId::DESCRIBED_MANUSCRIPT)&.first&.entity_id_value # P3
-    end
-
-    def holding_status
-      claims_by_property_id(PropertyId::HOLDING_STATUS)&.first&.entity_id_value # P6
-    end
-
-    def iiif_manifest
-      claims_by_property_id(PropertyId::IIIF_MANIFEST)&.first&.entity_id_value # P41
-    end
-
-    def core_model_item?
-      instance_of_claims.any? { |claim| ItemId::CORE_MODEL_ITEMS.include? claim.entity_id_value }
-    end
-
-    def manuscript?
-      instance_of_claims.any? { |claim| claim.entity_id_value == ItemId::MANUSCRIPT }
-    end
-
-    def holding?
-      instance_of_claims.any? { |claim| claim.entity_id_value == ItemId::HOLDING }
-    end
+    include PropertyId
 
-    def record?
-      instance_of_claims.any? { |claim| claim.entity_id_value == ItemId::RECORD }
+    def instance_of
+      claims_by_property_id(INSTANCE_OF)&.first&.entity_id_value
     end
   end
 end

data/lib/digital_scriptorium/ds_meta.rb

@@ -21,14 +21,8 @@ module DigitalScriptorium
       @record = record
     end
 
-    def current?(holding)
-      holding.holding_status == HOLDING_STATUS_CURRENT
-    end
-
     def current_holdings(manuscript, export_hash)
-      manuscript.holding_ids
-                .map { |id| export_hash[id] }
-                .filter { |holding| current?(holding) }
+      manuscript.holding_ids.filter_map { |id| export_hash[id] if export_hash[id]&.current? }
     end
   end
 end

data/lib/digital_scriptorium/export.rb

@@ -11,5 +11,12 @@ module DigitalScriptorium
       end
       hash
     end
+
+    def instance_of_id_from(item_hash)
+      claims = item_hash['claims']
+      return nil unless claims&.any?
+
+      claims.dig('P16', 0, 'mainsnak', 'datavalue', 'value', 'id')
+    end
   end
 end

data/lib/digital_scriptorium/export_representer.rb

@@ -6,6 +6,7 @@ require 'wikibase_representable'
 module DigitalScriptorium
   # Representer class for deserializing Wikibase data exports from JSON.
   class ExportRepresenter < Representable::Decorator
+    include ItemId
     include Representable::JSON::Collection
     include WikibaseRepresentable::Model
     include WikibaseRepresentable::Representers
@@ -13,7 +14,18 @@ module DigitalScriptorium
     items decorator: lambda { |input:, **|
       input.type == Item::ENTITY_TYPE ? ItemRepresenter : PropertyRepresenter
     }, class: lambda { |input:, **|
-      input['type'] == Item::ENTITY_TYPE ? DsItem : Property
+      return Property unless input['type'] == Item::ENTITY_TYPE
+
+      case instance_of_id_from input
+      when MANUSCRIPT
+        Manuscript
+      when HOLDING
+        Holding
+      when DS_20_RECORD
+        Record
+      else
+        DsItem
+      end
     }
   end
 end

data/lib/digital_scriptorium/holding.rb

@@ -3,32 +3,21 @@
 module DigitalScriptorium
   # An item representing a Digital Scriptorium holding (instance of Q2)
   class Holding < DsItem
-    def institution_as_recorded_claims
-      claims_by_property_id HOLDING_INSTITUTION_AS_RECORDED # P5
-    end
+    include ItemId
+    include PropertyId
 
     def status_claims
       claims_by_property_id HOLDING_STATUS # P6
     end
 
-    def institutional_id_claims
-      claims_by_property_id INSTITUTIONAL_ID # P7
-    end
-
-    def shelfmark_claims
-      claims_by_property_id SHELFMARK # P8
-    end
-
-    def link_to_institutional_record_claims
-      claims_by_property_id LINK_TO_INSTITUTIONAL_RECORD # P9
-    end
+    def status
+      return unless status_claims&.any?
 
-    def start_time_claims
-      claims_by_property_id START_TIME # P38
+      status_claims&.first&.entity_id_value
     end
 
-    def end_time_claims
-      claims_by_property_id END_TIME # P39
+    def current?
+      status == HOLDING_STATUS_CURRENT
     end
   end
 end

data/lib/digital_scriptorium/item_id.rb

@@ -5,11 +5,24 @@ require 'set'
 module DigitalScriptorium
   # Constants for core model item IDs.
   module ItemId
-    MANUSCRIPT             = 'Q1'
-    HOLDING                = 'Q2'
-    RECORD                 = 'Q3'
-    HOLDING_STATUS_CURRENT = 'Q4'
+    MANUSCRIPT                 = 'Q1'
+    HOLDING                    = 'Q2'
+    DS_20_RECORD               = 'Q3'
+    HOLDING_STATUS_CURRENT     = 'Q4'
+    HOLDING_STATUS_NON_CURRENT = 'Q5'
+    STANDARD_TITLE             = 'Q6'
+    ACTOR                      = 'Q7'
+    PERSONAL_NAME              = 'Q8'
+    CORPORATE_NAME             = 'Q9'
+    ROLE                       = 'Q10'
+    TERM                       = 'Q11'
+    LANGUAGE                   = 'Q12'
+    CENTURY                    = 'Q13'
+    DATED                      = 'Q14'
+    UNDATED                    = 'Q15'
+    PLACE                      = 'Q16'
+    MATERIAL                   = 'Q17'
 
-    CORE_MODEL_ITEMS = Set[MANUSCRIPT, HOLDING, RECORD]
+    CORE_MODEL_ITEMS = Set[MANUSCRIPT, HOLDING, DS_20_RECORD]
   end
 end

data/lib/digital_scriptorium/record.rb

@@ -8,65 +8,5 @@ module DigitalScriptorium
     def described_manuscript_id
       claims_by_property_id(DESCRIBED_MANUSCRIPT)&.first&.entity_id_value # P3
     end
-
-    def title_as_recorded_claims
-      claims_by_property_id TITLE_AS_RECORDED # P10
-    end
-
-    def uniform_title_as_recorded_claims
-      claims_by_property_id UNIFORM_TITLE_AS_RECORDED # P12
-    end
-
-    def associated_name_as_recorded_claims
-      claims_by_property_id ASSOCIATED_NAME_AS_RECORDED # P14
-    end
-
-    def genre_as_recorded_claims
-      claims_by_property_id GENRE_AS_RECORDED # P18
-    end
-
-    def language_as_recorded_claims
-      claims_by_property_id LANGUAGE_AS_RECORDED # P21
-    end
-
-    def production_date_as_recorded_claims
-      claims_by_property_id PRODUCTION_DATE_AS_RECORDED # P23
-    end
-
-    def dated_claims
-      claims_by_property_id DATED # P26
-    end
-
-    def production_place_as_recorded_claims
-      claims_by_property_id PRODUCTION_PLACE_AS_RECORDED # P27
-    end
-
-    def physical_description_claims
-      claims_by_property_id PHYSICAL_DESCRIPTION # P29
-    end
-
-    def material_as_recorded_claims
-      claims_by_property_id MATERIAL_AS_RECORDED # P30
-    end
-
-    def note_claims
-      claims_by_property_id NOTE # P32
-    end
-
-    def acknowledgements_claims
-      claims_by_property_id ACKNOWLEDGEMENTS # P33
-    end
-
-    def date_added_claims
-      claims_by_property_id DATE_ADDED # P34
-    end
-
-    def date_last_updated_claims
-      claims_by_property_id DATE_LAST_UPDATED # P35
-    end
-
-    def iiif_manifest_claims
-      claims_by_property_id IIIF_MANIFEST # P41
-    end
   end
 end

data/lib/digital_scriptorium/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DigitalScriptorium
-  VERSION = '0.2.1'
+  VERSION = '0.3.0'
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: digital_scriptorium
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Michael Holloway
 bindir: exe
 cert_chain: []
-date: 2025-01-18 00:00:00.000000000 Z
+date: 2025-02-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: wikibase_representable
@@ -37,6 +37,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.5'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.14'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement