search_solr_tools 6.4.0 → 6.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 310dba4a749b285756284729fe320c2344ac345c3251225dc1a46cc4c0f38d79
-  data.tar.gz: 98a397d862c73f27b9554560c55f2132f6d3211018c4cbe76a98f501cac2d5d0
+  metadata.gz: daf440444527ee68761274399910b02c51d7994a1b605c57f64b2666d7a243c7
+  data.tar.gz: 13a1a0ccd9d623aad334f68c6d314f7205a48a45546393d3d2ed4d71a95899cc
 SHA512:
-  metadata.gz: d02152bd10e9bc3ea59d6a2a8448ed2ab63da49db35dc0b23dbf27236f85253987169eaefbe2a3da3e527f9d63d11bdab08d58b3ebb9673579daebbdb8de6097
-  data.tar.gz: 3cc3e2199432b47dd02a1e45238c8405821673c808cd83392ebda9837890f949b193f9267b3afbd760f8449490ef5e97a911ad4c97dd856081307ba7f73095b4
+  metadata.gz: 2b195f73dc88f1fcf83b354f7ea572082419172970ced5e15e7f7afdb89f47da85424b473e9aa87462483f3ba6c5a2e1e2eb33c1493d72a978ff6570d86d9645
+  data.tar.gz: 96b72cadd2046588046b56a9911ee14caefead8813c38d579b93d9df56b00afacfeae13033c3ed5b866076c81aa694ccdf77fe024c354191c52d65db04c7f186

data/CHANGELOG.md

@@ -1,3 +1,15 @@
+## v6.5.0 (2023-09-21)
+
+  - Adding logging functionality to the code, including the ability
+    to specify log file destination and log level for both the file and
+    console output
+
+## v6.4.1 (2023-09-15)
+
+  - Added GitHub Action workflows for continuous integration features
+  - Updated bump rake task to use Bump gem
+  - Removed release rake task, moved it to the CI workflow
+
 ## v6.4.0 (2023-08-14)
 
   - Fixed a bug with the sanitization, which was trying to modify the

data/README.md

@@ -101,51 +101,79 @@ the tests whenever the appropriate files are changed.
 
 Please be sure to run them in the `bundle exec` context if you're utilizing bundler.
 
+By default, tests are run with minimal logging - no log file and only fatal errors
+written to the console.  This can be changed by setting the environment variables 
+as described in [Logging](#logging) below.
+
 ### Creating Releases (NSIDC devs only)
 
 Requirements:
 
 * Ruby > 3.2.2
 * [Bundler](http://bundler.io/)
-* [Gem Release](https://github.com/svenfuchs/gem-release)
 * [Rake](https://github.com/ruby/rake)
-* A [RubyGems](https://rubygems.org) account that has
-  [ownership](http://guides.rubygems.org/publishing/) of the gem
 * RuboCop and the unit tests should all pass (`rake`)
 
-The [CHANGELOG.md](CHANGELOG.md) is not automatically updated by the
-`rake release:*` tasks. Update it manually to insert the correct version and
-date, and commit the file, before creating the release package.
+To make a release, follow these steps:
 
-**gem release** is used by rake tasks in this project to handle version changes,
-tagging, and publishing to RubyGems.
+1. Confirm no errors are returned by `bundle exec rubocop` *
+2. Confirm all tests pass (`bundle exec rake spec:unit`) *
+3. Ensure that the `CHANGELOG.md` file is up to date with an `Unreleased`
+   header.
+4. Submit a Pull Request
+5. Once the PR has been reviewed and approved, merge the branch into `main`
+6. On your local machine, ensure you are on the `main` branch (and have
+   it up-to-date), and run `bundle exec rake bump:<part>` (see below)
+   * This will trigger the GitHub Actions CI workflow to push a release to
+     RubyGems.
 
-| Command                   | Description |
-|---------------------------|-------------|
-| `rake release:pre[false]` | Increase the current prerelease version number, push changes |
-| `rake release:pre[true]`  | Increase the current prerelease version number, publish release\* |
-| `rake release:none`       | Drop the prerelease version, publish release\*, then `pre[false]` (does a patch release) |
-| `rake release:minor`      | Increase the minor version number, publish release\*, then `pre[false]` |
-| `rake release:major`      | Increase the major version number, publish release\*, then `pre[false]` |
+The steps marked `*` above don't need to be done manually; every time a commit
+is pushed to the GitHub repository, these tests will be run automatically.
 
-\*"publish release" means each of the following occurs:
+The first 4 steps above are self-explanatory.  More information on the last
+steps can be found below.
 
-* a new tag is created
-* the changes are pushed
-* the tagged version is built and published to RubyGems
+#### Version Bumping
 
-You will need to have a current Rubygems API key for the _NSIDC developer user_ account in
-order to publish a new version of the gem to Rubygems. To get the lastest API key:
+Running the `bundle exec rake bump:<part>` tasks will do the following actions:
 
-`curl -u <username> https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials; chmod 0600 ~/.gem/credentials`
+1. The gem version will be updated locally
+2. The `CHANGELOG.md` file will updated with the updated gem version and date
+3. A tag `vx.x.x` will be created (with the new gem version)
+4. The files updated by the bump will be pushed to the GitHub repository, along
+   with the newly created tag.
+
+The sub-tasks associated with bump will allow the type of bump to be determined:
+
+| Command                   | Description                                                                                        |
+|---------------------------|----------------------------------------------------------------------------------------------------|
+| `rake bump:pre`           | Increase the current prerelease version number (v1.2.3 -> v1.2.3.pre1; v1.2.3.pre1 -> v1.2.3.pre2) |
+| `rake bump:patch`         | Increase the current patch number (v1.2.0 -> v1.2.1; v1.2.4 -> v1.2.4)                             |
+| `rake bump:minor`         | Increase the minor version number (v1.2.0 -> v1.3.0; v1.2.4 -> v1.3.0)                             |
+| `rake bump:major`         | Increase the major version number (v1.2.0 _> v2.0.0; v1.2.4 -> v2.0.0)                             |
+
+Using any bump other than `pre` will remove the `pre` suffix from the version as well.
+
+#### Release to RubyGems
 
-## Release steps (summary)
+When a tag in the format of `vx.y.z` (including a `pre` suffix) is pushed to GitHub,
+it will trigger the GitHub Actions release workflow.  This workflow will:
 
-- Confirm no errors are returned by `bundle exec rubocop`
-- Confirm all tests pass (`bundle exec rake spec:unit`)
-- Update the version number and date manually in `CHANGELOG.md` and commit the
-  changes.
-- Run the appropriate `bundle exec rake release:*` task
+1. Build the gem
+2. Push the gem to RubyGems
+
+The CI workflow has the credentials set up to push to RubyGems, so no user intervention
+is needed, and the workflow itself does not have to be manually triggered.
+
+If needed, the release can also be done locally by running the command
+`bundle exec gem release`. In order for this to work, you will need to have a
+local copy of current Rubygems API key for the _NSIDC developer user_ account in
+To get the lastest API key:
+
+`curl -u <username> https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials; chmod 0600 ~/.gem/credentials`
+
+It is recommended that this not be run locally, however; use the GitHub Actions CI
+workflow instead.
 
 ### SOLR
 
@@ -184,6 +212,47 @@ which can be modified, or additional environments can be added by just adding a
 new YAML stanza with the right keys; this new environment can then be used with
 the `--environment` flag when running `search_solr_tools harvest`.
 
+#### Logging
+
+By default, when running the harvest, harvest logs are written to the file
+`/var/log/search-solr-tools.log` (set to `warn` level), as well as to the console
+at `info` level.  These settings are configured in the `environments.yaml` config
+file, in the `common` section. 
+
+The keys in the `environments.yaml` file to consider are as follows:
+
+* `log_file` - The full name and path of the file to which log output will be written
+  to.  If set to the special value `none`, no log file will be written to at all.
+  Log output will be **appended** to the file, if it exists; otherwise, the file will
+  be created.
+* `log_file_level` - Indicates the level of logging which should be written to the log file.
+* `log_stdout_level` - Indicates the level of logging which should be written to the console.
+  This can be different than the level written to the log file.
+
+You can also override the configuration file settings at the command line with the
+following environment variables (useful when for doing development work):
+
+* `SEARCH_SOLR_LOG_FILE` - Overrides the `log_file` setting
+* `SEARCH_SOLR_LOG_LEVEL` - Overrides the `log_file_level` setting
+* `SEARCH_SOLR_STDOUT_LEVEL` - Overrides the `log_stdout_level` setting
+
+When running the spec tests, `SEARCH_SOLR_LOG_FILE` is set to `none` and
+`SEARCH_SOLR_STDOUT_LEVEL` is set to `fatal`, unless you manually set those
+environment variables prior to running the tests.  This is to keep the test output
+clean unless you need more detail for debugging.
+
+The following are the levels of logging that can be specified.  These levels are
+cumulative; for example, `error` will also output `fatal` log entries, and `debug`
+will output **all** log entries.
+
+* `none` - No logging outputs will be written.
+* `fatal` - Only outputs errors which result in a crash.
+* `error` - Outputs any error that occurs while harvesting.
+* `warn` - Outputs warnings that occur that do not cause issues with the harvesting,
+  but might indicate things that may need to be addressed (such as deprecations, etc)
+* `info` - Outputs general information, such as harvesting status
+* `debug` - Outputs detailed information that can be used for debugging and code tracing.
+
 ## Organization Info
 
 ### How to contact NSIDC

data/bin/search_solr_tools

@@ -6,8 +6,14 @@ require 'thor'
 
 # rubocop:disable Metrics/AbcSize
 class SolrHarvestCLI < Thor
+  include SSTLogger
+
   map %w[--version -v] => :__print_version
 
+  def self.exit_on_failure?
+    false
+  end
+
   desc '--version, -v', 'print the version'
   def __print_version
     puts SearchSolrTools::VERSION
@@ -39,10 +45,11 @@ class SolrHarvestCLI < Thor
       rescue StandardError => e
         solr_status = false
         source_status = false
-        puts "Error trying to ping for #{target}: #{e}"
+        logger.error "Ping failed for #{target}: #{e}}"
       end
       solr_success &&= solr_status
       source_success &&= source_status
+
       puts "Target: #{target}, Solr ping OK? #{solr_status}, data center ping OK? #{source_status}"
     end
 
@@ -61,7 +68,7 @@ class SolrHarvestCLI < Thor
   option :die_on_failure, type: :boolean
   def harvest(die_on_failure = options[:die_on_failure] || false)
     options[:data_center].each do |target|
-      puts "Target: #{target}"
+      logger.info "Target: #{target}"
       begin
         harvest_class = get_harvester_class(target)
         harvester = harvest_class.new(options[:environment], die_on_failure:)
@@ -73,12 +80,12 @@ class SolrHarvestCLI < Thor
 
         harvester.harvest_and_delete
       rescue SearchSolrTools::Errors::HarvestError => e
-        puts "THERE WERE HARVEST STATUS ERRORS:\n#{e.message}"
+        logger.error "THERE WERE HARVEST STATUS ERRORS:\n#{e.message}"
         exit e.exit_code
       rescue StandardError => e
         # If it gets here, there is an error that we aren't expecting.
-        puts "harvest failed for #{target}: #{e.message}"
-        puts e.backtrace
+        logger.error "harvest failed for #{target}: #{e.message}"
+        logger.error e.backtrace
         exit SearchSolrTools::Errors::HarvestError::ERRCODE_OTHER
       end
     end
@@ -93,16 +100,20 @@ class SolrHarvestCLI < Thor
   option :environment, required: true
   def delete_all
     env = SearchSolrTools::SolrEnvironments[options[:environment]]
+    logger.info('DELETE ALL started')
     `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<delete><query>*:*</query></delete>'`
     `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<commit/>'`
+    logger.info('DELETE ALL complete')
   end
 
   desc 'delete_all_auto_suggest', 'Delete all documents from the auto_suggest index'
   option :environment, required: true
   def delete_all_auto_suggest
     env = SearchSolrTools::SolrEnvironments[options[:environment]]
+    logger.info('DELETE ALL AUTO_SUGGEST started')
     `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<delete><query>*:*</query></delete>'`
     `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<commit/>'`
+    logger.info('DELETE ALL AUTO_SUGGEST complete')
   end
 
   desc 'delete_by_data_center', 'Force deletion of documents for a specific data center with timestamps before the passed timestamp in format iso8601 (2014-07-14T21:49:21Z)'
@@ -110,11 +121,13 @@ class SolrHarvestCLI < Thor
   option :environment, required: true
   option :data_center, required: true
   def delete_by_data_center
+    logger.info("DELETE ALL for data center '#{options[:data_center]}' started")
     harvester = get_harvester_class(options[:data_center]).new options[:environment]
     harvester.delete_old_documents(options[:timestamp],
                                    "data_centers:\"#{SearchSolrTools::Helpers::SolrFormat::DATA_CENTER_NAMES[options[:data_center].upcase.to_sym][:long_name]}\"",
                                    SearchSolrTools::SolrEnvironments[harvester.environment][:collection_name],
                                    true)
+    logger.info("DELETE ALL for data center '#{options[:data_center]}' complete")
   end
 
   no_tasks do

data/lib/search_solr_tools/config/environments.rb

@@ -5,10 +5,10 @@ require 'yaml'
 module SearchSolrTools
   # configuration to work with solr locally, or on integration/qa/staging/prod
   module SolrEnvironments
-    YAML_ENVS = YAML.load_file(File.expand_path('environments.yaml', __dir__))
+    YAML_ENVS = YAML.load_file(File.expand_path('environments.yaml', __dir__), aliases: true)
 
     def self.[](env = :development)
-      YAML_ENVS[:common].merge(YAML_ENVS[env.to_sym])
+      YAML_ENVS[env.to_sym]
     end
   end
 end

data/lib/search_solr_tools/config/environments.yaml

@@ -1,4 +1,4 @@
-:common:
+:common: &common
   :auto_suggest_collection_name: auto_suggest
   :collection_name: nsidc_oai
   :collection_path: solr
@@ -9,33 +9,48 @@
   # should be. GLA01.018 will show up if we use DCS API v2.
   :nsidc_oai_identifiers_url: oai?verb=ListIdentifiers&metadataPrefix=dif&retired=false
 
+  # Log details.  Can be overridden by environment-specific values
+  :log_file: /var/log/search-solr-tools.log
+  :log_file_level: warn
+  :log_stdout_level: info
+
 :local:
+  <<: *common
   :host: localhost
   :nsidc_dataset_metadata_url: http://integration.nsidc.org/api/dataset/metadata/
 
-:dev:
+:dev: &dev
+  <<: *common
   ## For the below, you'll need to instantiate your own search-solr instance, and point host to that.
   :host: dev.search-solr.USERNAME.dev.int.nsidc.org
   ## For the metadata content, either set up your own instance of dataset-catalog-services
   ## or change the URL below to point to integration
   :nsidc_dataset_metadata_url: http://dev.dcs.USERNAME.dev.int.nsidc.org:1580/api/dataset/metadata/
 
+:development:
+  <<: *dev
+
 :integration:
+  <<: *common
   :host: integration.search-solr.apps.int.nsidc.org
   :nsidc_dataset_metadata_url: http://integration.nsidc.org/api/dataset/metadata/
 
 :qa:
+  <<: *common
   :host: qa.search-solr.apps.int.nsidc.org
   :nsidc_dataset_metadata_url: http://qa.nsidc.org/api/dataset/metadata/
 
 :staging:
+  <<: *common
   :host: staging.search-solr.apps.int.nsidc.org
   :nsidc_dataset_metadata_url: http://staging.nsidc.org/api/dataset/metadata/
 
 :blue:
+  <<: *common
   :host: blue.search-solr.apps.int.nsidc.org
   :nsidc_dataset_metadata_url: http://nsidc.org/api/dataset/metadata/
 
 :production:
+  <<: *common
   :host: search-solr.apps.int.nsidc.org
   :nsidc_dataset_metadata_url: http://nsidc.org/api/dataset/metadata/

data/lib/search_solr_tools/errors/harvest_error.rb

@@ -1,8 +1,12 @@
 # frozen_string_literal: true
 
+require_relative '../logging/sst_logger'
+
 module SearchSolrTools
   module Errors
     class HarvestError < StandardError
+      include SSTLogger
+
       ERRCODE_SOLR_PING = 1
       ERRCODE_SOURCE_PING = 2
       ERRCODE_SOURCE_NO_RESULTS = 4
@@ -73,11 +77,11 @@ module SearchSolrTools
       # rubocop:disable Metrics/AbcSize
       def exit_code
         if @status_data.nil?
-          puts "OTHER ERROR REPORTED: #{@other_message}"
+          logger.error "OTHER ERROR REPORTED: #{@other_message}"
           return ERRCODE_OTHER
         end
 
-        puts "EXIT CODE STATUS:\n#{@status_data.status}"
+        logger.error "EXIT CODE STATUS:\n#{@status_data.status}"
 
         code = 0
         code += ERRCODE_SOLR_PING unless @status_data.ping_solr

data/lib/search_solr_tools/harvesters/auto_suggest.rb

@@ -51,10 +51,10 @@ module SearchSolrTools
         status = insert_solr_doc add_docs, Base::JSON_CONTENT_TYPE, @env_settings[:auto_suggest_collection_name]
 
         if status == Helpers::HarvestStatus::INGEST_OK
-          puts "Added #{add_docs.size} auto suggest documents in one commit"
+          logger.info "Added #{add_docs.size} auto suggest documents in one commit"
           Helpers::HarvestStatus.new(Helpers::HarvestStatus::INGEST_OK => add_docs)
         else
-          puts "Failed adding #{add_docs.size} documents in single commit, retrying one by one"
+          logger.error "Failed adding #{add_docs.size} documents in single commit, retrying one by one"
           new_add_docs = []
           add_docs.each do |doc|
             new_add_docs << { 'add' => { 'doc' => doc } }

data/lib/search_solr_tools/harvesters/base.rb

@@ -15,6 +15,8 @@ module SearchSolrTools
   module Harvesters
     # base class for solr harvesters
     class Base
+      include SSTLogger
+
       attr_accessor :environment
 
       DELETE_DOCUMENTS_RATIO = 0.1
@@ -50,10 +52,10 @@ module SearchSolrTools
         begin
           RestClient.get(url) do |response, _request, _result|
             success = response.code == 200
-            puts "Error in ping request: #{response.body}" unless success
+            logger.error "Error in ping request: #{response.body}" unless success
           end
         rescue StandardError => e
-          puts "Rest exception while pinging Solr: #{e}"
+          logger.error "Rest exception while pinging Solr: #{e}"
         end
         success
       end
@@ -62,7 +64,7 @@ module SearchSolrTools
       # to "ping" the data center.  Returns true if the ping is successful (or, as
       # in this default, no ping method was defined)
       def ping_source
-        puts 'Harvester does not have ping method defined, assuming true'
+        logger.info 'Harvester does not have ping method defined, assuming true'
         true
       end
 
@@ -81,9 +83,9 @@ module SearchSolrTools
         solr = RSolr.connect url: solr_url + "/#{solr_core}"
         unchanged_count = (solr.get 'select', params: { wt: :ruby, q: delete_query, rows: 0 })['response']['numFound'].to_i
         if unchanged_count.zero?
-          puts "All documents were updated after #{timestamp}, nothing to delete"
+          logger.info "All documents were updated after #{timestamp}, nothing to delete"
         else
-          puts "Begin removing documents older than #{timestamp}"
+          logger.info "Begin removing documents older than #{timestamp}"
           remove_documents(solr, delete_query, constraints, force, unchanged_count)
         end
       end
@@ -99,13 +101,13 @@ module SearchSolrTools
       def remove_documents(solr, delete_query, constraints, force, numfound)
         all_response_count = (solr.get 'select', params: { wt: :ruby, q: constraints, rows: 0 })['response']['numFound']
         if force || (numfound / all_response_count.to_f < DELETE_DOCUMENTS_RATIO)
-          puts "Deleting #{numfound} documents for #{constraints}"
+          logger.info "Deleting #{numfound} documents for #{constraints}"
           solr.delete_by_query delete_query
           solr.commit
         else
-          puts "Failed to delete records older than current harvest start because they exceeded #{DELETE_DOCUMENTS_RATIO} of the total records for this data center."
-          puts "\tTotal records: #{all_response_count}"
-          puts "\tNon-updated records: #{numfound}"
+          logger.info "Failed to delete records older than current harvest start because they exceeded #{DELETE_DOCUMENTS_RATIO} of the total records for this data center."
+          logger.info "\tTotal records: #{all_response_count}"
+          logger.info "\tNon-updated records: #{numfound}"
         end
       end
 
@@ -121,8 +123,8 @@ module SearchSolrTools
           status.record_status doc_status
           doc_status == Helpers::HarvestStatus::INGEST_OK ? success += 1 : failure += 1
         end
-        puts "#{success} document#{success == 1 ? '' : 's'} successfully added to Solr."
-        puts "#{failure} document#{failure == 1 ? '' : 's'} not added to Solr."
+        logger.info "#{success} document#{success == 1 ? '' : 's'} successfully added to Solr."
+        logger.info "#{failure} document#{failure == 1 ? '' : 's'} not added to Solr."
 
         status
       end
@@ -146,14 +148,14 @@ module SearchSolrTools
           RestClient.post(url, doc_serialized, content_type:) do |response, _request, _result|
             success = response.code == 200
             unless success
-              puts "Error for #{doc_serialized}\n\n response: #{response.body}"
+              logger.error "Error for #{doc_serialized}\n\n response: #{response.body}"
               status = Helpers::HarvestStatus::INGEST_ERR_SOLR_ERROR
             end
           end
         rescue StandardError => e
           # TODO: Need to provide more detail re: this failure so we know whether to
           #  exit the job with a status != 0
-          puts "Rest exception while POSTing to Solr: #{e}, for doc: #{doc_serialized}"
+          logger.error "Rest exception while POSTing to Solr: #{e}, for doc: #{doc_serialized}"
           status = Helpers::HarvestStatus::INGEST_ERR_SOLR_ERROR
         end
         status
@@ -177,11 +179,11 @@ module SearchSolrTools
         request_url = encode_data_provider_url(request_url)
 
         begin
-          puts "Request: #{request_url}"
+          logger.debug "Request: #{request_url}"
           response = URI.parse(request_url).open(read_timeout: timeout, 'Content-Type' => content_type)
         rescue OpenURI::HTTPError, Timeout::Error, Errno::ETIMEDOUT => e
           retries_left -= 1
-          puts "## REQUEST FAILED ## #{e.class} ## Retrying #{retries_left} more times..."
+          logger.error "## REQUEST FAILED ## #{e.class} ## Retrying #{retries_left} more times..."
 
           retry if retries_left.positive?
 

data/lib/search_solr_tools/harvesters/nsidc_auto_suggest.rb

@@ -6,7 +6,7 @@ module SearchSolrTools
   module Harvesters
     class NsidcAutoSuggest < AutoSuggest
       def harvest_and_delete
-        puts 'Building auto-suggest indexes for NSIDC'
+        logger.info 'Building auto-suggest indexes for NSIDC'
         super(method(:harvest), 'source:"NSIDC"', @env_settings[:auto_suggest_collection_name])
       end
 

data/lib/search_solr_tools/harvesters/nsidc_json.rb

@@ -21,13 +21,13 @@ module SearchSolrTools
             return response.code == 200
           end
         rescue StandardError
-          puts "Error trying to get options for #{nsidc_json_url} (ping)"
+          logger.error "Error trying to get options for #{nsidc_json_url} (ping)"
         end
         false
       end
 
       def harvest_and_delete
-        puts "Running harvest of NSIDC catalog from #{nsidc_json_url}"
+        logger.info "Running harvest of NSIDC catalog from #{nsidc_json_url}"
         super(method(:harvest_nsidc_json_into_solr), "data_centers:\"#{Helpers::SolrFormat::DATA_CENTER_NAMES[:NSIDC][:long_name]}\"")
       end
 
@@ -47,8 +47,8 @@ module SearchSolrTools
       rescue Errors::HarvestError => e
         raise e
       rescue StandardError => e
-        puts "An unexpected exception occurred while trying to harvest or insert: #{e}"
-        puts e.backtrace
+        logger.error "An unexpected exception occurred while trying to harvest or insert: #{e}"
+        logger.error e.backtrace
         status = Helpers::HarvestStatus.new(Helpers::HarvestStatus::OTHER_ERROR => e)
         raise Errors::HarvestError, status
       end
@@ -83,7 +83,7 @@ module SearchSolrTools
           begin
             docs << { 'add' => { 'doc' => @translator.translate(fetch_json_from_nsidc(id)) } }
           rescue StandardError => e
-            puts "Failed to fetch #{id} with error #{e}: #{e.backtrace}"
+            logger.error "Failed to fetch #{id} with error #{e}: #{e.backtrace}"
             failure_ids << id
           end
         end

data/lib/search_solr_tools/logging/sst_logger.rb

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'logging'
+
+require_relative '../config/environments'
+
+module SSTLogger
+  LOG_LEVELS = %w[debug info warn error fatal none].freeze
+
+  def logger
+    SSTLogger.logger
+  end
+
+  class << self
+    def logger
+      @logger ||= new_logger
+    end
+
+    private
+
+    def new_logger
+      logger = Logging.logger['search_solr_tools']
+
+      append_stdout_logger(logger)
+      append_file_logger(logger)
+
+      logger
+    end
+
+    def append_stdout_logger(logger)
+      return if log_stdout_level.nil?
+
+      new_stdout = Logging.appenders.stdout
+      new_stdout.level = log_stdout_level
+      new_stdout.layout = Logging.layouts.pattern(pattern: "%-5l : %m\n")
+      logger.add_appenders new_stdout
+    end
+
+    def append_file_logger(logger)
+      return if log_file == 'none'
+
+      FileUtils.mkdir_p(File.dirname(log_file))
+      new_file = Logging.appenders.file(
+        log_file,
+        layout: Logging.layouts.pattern(pattern: "[%d] %-5l : %m\n")
+      )
+      new_file.level = log_file_level
+      logger.add_appenders new_file
+    end
+
+    def log_file
+      env = SearchSolrTools::SolrEnvironments[]
+      ENV.fetch('SEARCH_SOLR_LOG_FILE', nil) || env[:log_file]
+    end
+
+    def log_file_level
+      env = SearchSolrTools::SolrEnvironments[]
+      log_level(ENV.fetch('SEARCH_SOLR_LOG_LEVEL', nil) || env[:log_file_level])
+    end
+
+    def log_stdout_level
+      env = SearchSolrTools::SolrEnvironments[]
+      log_level(ENV.fetch('SEARCH_SOLR_STDOUT_LEVEL', nil) || env[:log_stdout_level])
+    end
+
+    def log_level(level)
+      LOG_LEVELS.include?(level) ? level : nil
+    end
+  end
+end

data/lib/search_solr_tools/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SearchSolrTools
-  VERSION = '6.4.0'
+  VERSION = '6.5.0'
 end

data/lib/search_solr_tools.rb

@@ -6,6 +6,8 @@ require_relative 'search_solr_tools/version'
 require_relative 'search_solr_tools/helpers/harvest_status'
 require_relative 'search_solr_tools/errors/harvest_error'
 
+require_relative 'search_solr_tools/logging/sst_logger'
+
 %w[harvesters translators].each do |subdir|
   Dir[File.join(__dir__, 'search_solr_tools', subdir, '*.rb')].each { |file| require file }
 end

data/search_solr_tools.gemspec

@@ -27,14 +27,16 @@ Gem::Specification.new do |spec|
 
   spec.add_runtime_dependency 'ffi-geos', '~> 2.4.0'
   spec.add_runtime_dependency 'iso8601', '~> 0.13.0'
+  spec.add_runtime_dependency 'logging', '~> 2.3.1'
   spec.add_runtime_dependency 'multi_json', '~> 1.15.0'
-  spec.add_runtime_dependency 'nokogiri', '~> 1.15.3'
+  spec.add_runtime_dependency 'nokogiri', '~> 1.15.4'
   spec.add_runtime_dependency 'rest-client', '~> 2.1.0'
   spec.add_runtime_dependency 'rgeo', '~> 3.0.0'
   spec.add_runtime_dependency 'rgeo-geojson', '~> 2.1.1'
   spec.add_runtime_dependency 'rsolr', '~> 2.5.0'
   spec.add_runtime_dependency 'thor', '~> 1.2.2'
 
+  spec.add_development_dependency 'bump', '~> 0.10.0'
   spec.add_development_dependency 'gem-release', '~> 2.2.2'
   spec.add_development_dependency 'guard', '~> 2.18.0'
   spec.add_development_dependency 'guard-rspec', '~> 4.7.3'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: search_solr_tools
 version: !ruby/object:Gem::Version
-  version: 6.4.0
+  version: 6.5.0
 platform: ruby
 authors:
 - Chris Chalstrom
@@ -14,7 +14,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2023-08-14 00:00:00.000000000 Z
+date: 2023-09-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ffi-geos
@@ -44,6 +44,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.13.0
+- !ruby/object:Gem::Dependency
+  name: logging
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.3.1
 - !ruby/object:Gem::Dependency
   name: multi_json
   requirement: !ruby/object:Gem::Requirement
@@ -64,14 +78,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.15.3
+        version: 1.15.4
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.15.3
+        version: 1.15.4
 - !ruby/object:Gem::Dependency
   name: rest-client
   requirement: !ruby/object:Gem::Requirement
@@ -142,6 +156,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 1.2.2
+- !ruby/object:Gem::Dependency
+  name: bump
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.10.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.10.0
 - !ruby/object:Gem::Dependency
   name: gem-release
   requirement: !ruby/object:Gem::Requirement
@@ -332,6 +360,7 @@ files:
 - lib/search_solr_tools/helpers/solr_format.rb
 - lib/search_solr_tools/helpers/translate_spatial_coverage.rb
 - lib/search_solr_tools/helpers/translate_temporal_coverage.rb
+- lib/search_solr_tools/logging/sst_logger.rb
 - lib/search_solr_tools/translators/nsidc_json.rb
 - lib/search_solr_tools/version.rb
 - search_solr_tools.gemspec
@@ -354,7 +383,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.17
+rubygems_version: 3.4.10
 signing_key: 
 specification_version: 4
 summary: Tools to harvest and manage various scientific dataset feeds in a Solr instance.