search_solr_tools 7.1.0 → 7.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 06c1b12c76f3c9cfdd99f05f85cce8a24c6118cd1dfcbfc8ace538a32865d36f
-  data.tar.gz: bd22205a6833f05d4d96fa43cdc91b5b3d45452f658ccf597ac3bbd26ac40abc
+  metadata.gz: ea0e97fdb6cd3986683da13d8fd8ff26a150b00a3fc3b64de54fee93d3ceaf1d
+  data.tar.gz: 9e0c10993ab2fce13b400f9fb419788f0706b4367d9629145e425ed8c5931d05
 SHA512:
-  metadata.gz: 6df6573622d4643676f1334b15aa4184f15403bc8f90f8a8cd116c25aa86202c6edc04a8acb9ef4ae428e49bb5429b2fad5b1ecd1afbf5bf5557af38eca28285
-  data.tar.gz: e75b60f41f1c4e5f8b2e7ad14e34a58fa20ba3af5d45b56fccf6914db51f108d4c015a26a139d21ed30c2cd9c06634f73c5a637043f336a210ca37f00625877c
+  metadata.gz: 9fa45abe863da34e10f81183d242db3ccf8e1b37b01b2efb0836f469f50ae1f36dc3cd03ca6ab27a0ea2d0a83db232b93bb98271b8948b3283f717f2fd03ee66
+  data.tar.gz: 190b02a7ecd6ed44c762317d4e142e88877f053ad09457187e05c5a9e6de7e41f307118ad64661cc8efe563d0bb23054d795c3aa0a307b3b71df4d764c30fd53

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## v7.2.0 (2023-10-19)
+
+  - Force parameter facets based on GCMD keywords to be upper-case.
+  - Only use short name for sensor facets in which the short name and long name are identical.
+
 ## v7.1.0 (2023-10-11)
 
   - Updating harvesting to harvest storage system and spatial coverage
@@ -50,15 +55,15 @@
 
 ## v5.2.0 (2022-08-31)
 
-  - Updated the call for identifiers for the json harvester to use the 
+  - Updated the call for identifiers for the json harvester to use the
     proper "metadataPrefix" parameter, and request the dif identifiers
     instead of iso.
 
 ## v5.1.0 (2020-07-23)
 
-  - Added a CLI method to "ping" the Solr and Source servers for a given 
+  - Added a CLI method to "ping" the Solr and Source servers for a given
     data center.
-  - Added a CLI method "errcode" to get information about the various 
+  - Added a CLI method "errcode" to get information about the various
     error codes that may be returned during harvest
   - Updated the CLI harvest to return more useful error codes on failure.
 

data/README.md

@@ -4,7 +4,7 @@
 
 This is a gem that contains:
 
-* Ruby translators to transform various metadata feeds into solr documents
+* Ruby translators to transform NSIDC metadata feeds into solr documents
 * A command-line utility to access/utilize the gem's translators to harvest
    metadata into a working solr instance.
 
@@ -25,30 +25,23 @@ Clone the repository, and install all requirements as noted below.
 #### Configuration
 
 Once you have the code and requirements, edit the configuration file in
-`lib/search_solr_tools/config/environments.yaml` to match your environment. The
-configuration values are set by environment for each harvester (or specified in
-the `common` settings list), with the environment overriding `common` if a
-different setting is specified for a given environment.
-
-Each harvester has its own configuration settings. Most are the target endpoint;
-EOL, however, has a list of THREDDS project endpoints and NSIDC has its own
-oai/metadata endpoint settings.
-
-Most users should not need to change the harvester configuration unless they
-establish a local test node, or if a provider changes available endpoints;
-however, the `host` option for each environment must specify the configured SOLR
+`lib/search_solr_tools/config/environments.yaml` to match your environment. 
+Environment settings take precedence over `common` settings.
+The `host` option for each environment must specify the configured SOLR
 instance you intend to use these tools with.
 
 #### Build and Install Gem
 
-Then run:
+Run:
 
   `bundle exec gem build ./search_solr_tools.gemspec`
 
-Once you have the gem built in the project directory, install the utility:
+Once you have the gem built in the project directory, install it:
 
   `gem install --local ./search_solr_tools-version.gem`
 
+See _Harvesting Data_ (below) for usage examples.
+
 ## Working on the Project
 
 1. Create your feature branch (`git checkout -b my-new-feature`)
@@ -177,8 +170,7 @@ workflow instead.
 
 ### SOLR
 
-To harvest data utilizing the gem, you will need an installed instance of [Solr
-8.5.3](https://lucene.apache.org/solr/guide/)
+To harvest data utilizing the gem, you will need an installed instance of [Solr](https://solr.apache.org/guide/solr/latest/index.html)
 
 #### NSIDC
 
@@ -193,7 +185,7 @@ Outside of NSIDC, setup solr using the instructions found in the
 
 ### Harvesting Data
 
-The harvester requires additional metadata from services that may not yet be
+The harvester requires additional metadata from services that may not be
 publicly available, which are referenced in
 `lib/search_solr_tools/config/environments.yaml`.
 
@@ -204,7 +196,7 @@ overview of what's available, simply run `search_solr_tools`.
 
 Harvesting of data can be done using the `harvest` task, giving it a list of
 harvesters and an environment. Deletion is possible via the `delete_all` and/or
-`delete_by_data_center'`tasks. `list harvesters` will list the valid harvest
+`delete_by_data_center'`tasks. `list_harvesters` will list the valid harvest
 targets.
 
 In addition to feed URLs, `environments.yaml` also defines various environments
@@ -212,6 +204,13 @@ which can be modified, or additional environments can be added by just adding a
 new YAML stanza with the right keys; this new environment can then be used with
 the `--environment` flag when running `search_solr_tools harvest`.
 
+An example harvest of NSIDC metadata into a developer instance of Solr:
+
+    bundle exec search_solr_tools harvest --data-center=nsidc --environment=dev
+
+In this example, the `host` value in the `environments.yaml` `dev` entry
+must reference a valid Solr instance.
+
 #### Logging
 
 By default, when running the harvest, harvest logs are written to the file

data/lib/search_solr_tools/harvesters/nsidc_json.rb

@@ -40,7 +40,7 @@ module SearchSolrTools
 
         status.record_status(Helpers::HarvestStatus::HARVEST_NO_DOCS) if (result[:num_docs]).zero?
 
-        # Record the number of harvest failures; note that if this is 0, thats OK, the status will stay at 0
+        # Record the number of harvest failures; note that if this is 0, that's OK, the status will stay at 0
         status.record_status(Helpers::HarvestStatus::HARVEST_FAILURE, result[:failure_ids].length)
 
         raise Errors::HarvestError, status unless status.ok?

data/lib/search_solr_tools/helpers/solr_format.rb

@@ -89,7 +89,7 @@ module SearchSolrTools
         binned_facet = bin(FacetConfiguration.get_facet_bin(type), format_string)
         if binned_facet.nil?
           format_string
-        elsif binned_facet.eql?('exclude')
+        elsif binned_facet.match?(/\Aexclude\z/i)
           nil
         else
           binned_facet
@@ -98,11 +98,14 @@ module SearchSolrTools
 
       def self.parameter_binning(parameter_string)
         binned_parameter = bin(FacetConfiguration.get_facet_bin('parameter'), parameter_string)
-        # use variable_level_1 if no mapping exists
         return binned_parameter unless binned_parameter.nil?
 
+        # if no mapping exists, use variable_level_1.
+        # Force it to all upper case for consistency. This is a hacky workaround to
+        # deal with deprecated GCMD keywords still in use by some datasets that result
+        # in duplicate, case-sensitive entries in the search interface facet list.
         parts = parameter_string.split '>'
-        return parts[3].strip if parts.length >= 4
+        return parts[3].strip.upcase if parts.length >= 4
 
         nil
       end
@@ -158,7 +161,7 @@ module SearchSolrTools
       def self.bin(mappings, term)
         mappings.each do |mapping|
           term.match(mapping['pattern']) do
-            return mapping['mapping']
+            return mapping['mapping'].upcase
           end
         end
         nil

data/lib/search_solr_tools/translators/nsidc_json.rb

@@ -75,9 +75,10 @@ module SearchSolrTools
         return facet_values if json.nil?
 
         json.each do |json_entry|
+          long_name = json_entry['shortName'].eql?(json_entry['longName']) ? '' : json_entry['longName']
           sensor_bin = Helpers::SolrFormat.facet_binning('sensor', json_entry['shortName'].to_s)
           facet_values << if sensor_bin.eql? json_entry['shortName']
-                            "#{json_entry['longName']} | #{json_entry['shortName']}"
+                            "#{long_name} | #{json_entry['shortName']}"
                           else
                             " | #{sensor_bin}"
                           end

data/lib/search_solr_tools/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SearchSolrTools
-  VERSION = '7.1.0'
+  VERSION = '7.2.0'
 end

data/search_solr_tools.gemspec

@@ -8,7 +8,7 @@ Gem::Specification.new do |spec|
   spec.name          = 'search_solr_tools'
   spec.version       = SearchSolrTools::VERSION
   spec.authors       = ['Chris Chalstrom', 'Michael Brandt', 'Jonathan Kovarik', 'Luis Lopez', 'Stuart Reed', 'Julia Collins', 'Scott Lewis']
-  spec.email         = ['cchalstr@nsidc.org', 'mbrandt@colorado.edu', 'kovarik@nsidc.org', 'luis.lopezespinosa@colorado.edu', 'stuart.reed@colorado.edu', 'jcollins@nsidc.org', 'scott.lewis@nsidc.org']
+  spec.email         = ['Jonathan.Kovarik@colorado.edu', 'luis.lopezespinosa@colorado.edu', 'collinsj@colorado.edu', 'scott.lewis@colorado.edu']
   spec.summary       = 'Tools to harvest and manage various scientific dataset feeds in a Solr instance.'
   spec.description   = <<-EOF
     Ruby translators to transform various metadata feeds into solr documents and

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: search_solr_tools
 version: !ruby/object:Gem::Version
-  version: 7.1.0
+  version: 7.2.0
 platform: ruby
 authors:
 - Chris Chalstrom
@@ -14,7 +14,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2023-10-11 00:00:00.000000000 Z
+date: 2023-10-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ffi-geos
@@ -329,13 +329,10 @@ description: |2
       a command-line utility to access/utilize the gem's translators to harvest
       metadata into a working solr instance.
 email:
-- cchalstr@nsidc.org
-- mbrandt@colorado.edu
-- kovarik@nsidc.org
+- Jonathan.Kovarik@colorado.edu
 - luis.lopezespinosa@colorado.edu
-- stuart.reed@colorado.edu
-- jcollins@nsidc.org
-- scott.lewis@nsidc.org
+- collinsj@colorado.edu
+- scott.lewis@colorado.edu
 executables:
 - search_solr_tools
 extensions: []