search_solr_tools 3.1.2

data/README.md

@@ -0,0 +1,203 @@
+[![Build Status](https://travis-ci.org/nsidc/search-solr-tools.svg?branch=master)](https://travis-ci.org/nsidc/search-solr-tools)
+
+# NSIDC Search Solr Tools
+
+This is a gem that contains:
+
+* Ruby translators to transform various metadata feeds into solr documents
+* A command-line utility to access/utilize the gem's translators to harvest
+   metadata into a working solr instance.
+
+## Using the project
+
+### Standard Installation
+
+The gem is available through [RubyGems](https://rubygems.org/). To install the
+gem, ensure all requirements below are met and run (providing the appropriate
+version):
+
+`sudo gem install search_solr_tools -v $VERSION`
+
+### Custom Deployment
+
+Clone the repository, and install all requirements as noted below.
+
+#### Configuration
+
+Once you have the code and requirements, edit the configuration file in
+`lib/search_solr_tools/config/environments.yaml` to match your environment. The
+configuration values are set by environment for each harvester (or specified in
+the `common` settings list), with the environment overriding `common` if a
+different setting is specified for a given environment.
+
+Each harvester has its own configuration settings. Most are the target endpoint;
+EOL, however, has a list of THREDDS project endpoints and NSIDC has its own
+oai/metadata endpoint settings.
+
+Most users should not need to change the harvester configuration unless they
+establish a local test node, or if a provider changes available endpoints;
+however, the `host` option for each environment must specify the configured SOLR
+instance you intend to use these tools with.
+
+#### Build and Install Gem
+
+Then run:
+
+  `bundle exec gem build ./search_solr_tools.gemspec`
+
+Once you have the gem built in the project directory, install the utility:
+
+  `gem install --local ./search_solr_tools-version.gem`
+
+## Working on the Project
+
+1. Create your feature branch (`git checkout -b my-new-feature`)
+2. Stage your changes (`git add`)
+3. Commit your Rubocop compliant and test-passing changes with a
+   [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
+  (`git commit`)
+4. Push to the branch (`git push -u origin my-new-feature`)
+5. Create a new Pull Request
+
+### Requirements
+
+* Ruby > 2.0.0
+* [Bundler](http://bundler.io/)
+* Requirements for nokogiri:
+    * [libxml2/libxml2-dev](http://xmlsoft.org/)
+    * [zlibc](http://www.zlibc.linux.lu/)
+    * [zlib1g/zlib1g-dev](http://zlib.net/)
+    * Dependency build requirements:
+        * For Ubuntu/Debian, install the build-essential package.
+        * On the latest Fedora release installing the following will get you all of the requirements:
+
+              `yum groupinstall 'Development Tools'`
+
+              `yum install gcc-c++`
+
+        *Please note*:  If you are having difficulty installing Nokogiri please review the
+          Nokogiri [installation tutorial](http://www.nokogiri.org/tutorials/installing_nokogiri.html)
+
+* All gems installed (preferably using bundler: `bundle install`)
+* A running, configured SOLR instance to accept data harvests.
+
+### RuboCop
+
+The style checker [RuboCop](https://github.com/bbatsov/rubocop) can be run with
+`rubocop` or `bundle exec rake guard:rubocop`. The rake task will also watch for
+ruby files (.rb, .rake, Gemfile, Guardfile, Rakefile) to be changed, and run
+RuboCop on the changed files.
+
+`bundle exec rake guard` will automatically run the unit tests and RuboCop in
+one terminal window.
+
+RuboCop can be configured by modifying `.rubocop.yml`.
+
+Pushing with failing tests or RuboCop violations will cause the Jenkins build to
+break. Jenkins jobs to build and deploy this project are named
+"NSIDC_Search_SOLR_()…" and can be viewed under the
+[NSIDC Search tab](https://scm.nsidc.org/jenkins/view/NSIDC%20Search/).
+
+### Testing
+
+Unit tests can be run with `rspec`, `bundle exec rake spec:unit`, or `bundle
+exec rake guard:specs`.  Running the rake guard task will also automatically run
+the tests whenever the appropriate files are changed.
+
+Please be sure to run them in the `bundle exec` context if you're utilizing bundler.
+
+### Creating Releases (NSIDC devs only)
+
+Requirements:
+
+* Ruby > 2.0.0
+* [Bundler](http://bundler.io/)
+* [Gem Release](https://github.com/svenfuchs/gem-release)
+* [Rake](https://github.com/ruby/rake)
+* a [RubyGems](https://rubygems.org) account that has
+  [ownership](http://guides.rubygems.org/publishing/) of the gem
+* RuboCop and the unit tests should all pass (`rake`)
+
+**gem release** is used by rake tasks in this project to handle version changes,
+tagging, and publishing to RubyGems.
+
+| Command                   | Description |
+|---------------------------|-------------|
+| `rake release:pre[false]` | Increase the current prerelease version number, push changes |
+| `rake release:pre[true]`  | Increase the current prerelease version number, publish release\* |
+| `rake release:none`       | Drop the prerelease version, publish release, then `pre[false]` |
+| `rake release:minor`      | Increase the minor version number, publish release, then `pre[false]` |
+| `rake release:major`      | Increase the major version number, publish release, then `pre[false]` |
+
+\*"publish release" means each of the following occurs:
+
+* a new tag is created
+* the changes are pushed
+* the tagged version is built and published to RubyGems
+
+### SOLR:
+
+To harvest data utilizing the gem, you will need a local configured instance of
+Solr 4.3, which can be downloaded from
+[Apache's archive](https://archive.apache.org/dist/lucene/solr/4.3.0/).
+
+#### NSIDC
+
+At NSIDC the development VM can be provisioned with the
+[solr puppet module](https://bitbucket.org/nsidc/puppet-solr/) to install and
+configure Solr.
+
+#### Non-NSIDC
+
+Outside of NSIDC, setup solr using the instructions found in the
+[search-solr](https://github.com/nsidc/search-solr) project.
+
+### Harvesting Data
+
+The harvester requires additional metadata from services that may not yet be
+publicly available, which are referenced in
+`lib/search_solr_tools/config/environments.yaml`.
+
+To utilize the gem, build and install the **search_solr_tools** gem. This will
+add an executable `search_solr_tools` to the path (source is in
+`bin/search_solr_tools`). The executable is self-documenting; for a brief
+overview of what's available, simply run `search_solr_tools`.
+
+Harvesting of data can be done using the `harvest` task, giving it a list of
+harvesters and an environment. Deletion is possible via the `delete_all` and/or
+`delete_by_data_center'`tasks. `list harvesters` will list the valid harvest
+targets.
+
+In addition to feed URLs, `environments.yaml` also defines various environments
+which can be modified, or additional environments can be added by just adding a
+new YAML stanza with the right keys; this new environment can then be used with
+the `--environment` flag when running `search_solr_tools harvest`.
+
+## Organization Info
+
+### How to contact NSIDC
+
+User Services and general information:  
+Support: [http://support.nsidc.org](http://support.nsidc.org)  
+Email: nsidc@nsidc.org
+
+Phone: +1 303.492.6199  
+Fax: +1 303.492.2468
+
+Mailing address:  
+National Snow and Ice Data Center  
+CIRES, 449 UCB  
+University of Colorado  
+Boulder, CO 80309-0449 USA
+
+### License
+
+Every file in this repository is covered by the GNU GPL Version 3; a copy of the
+license is included in the file COPYING.
+
+### Citation Information
+
+Andy Grauch, Brendan Billingsley, Chris Chalstrom, Danielle Harper, Ian
+Truslove, Jonathan Kovarik, Luis Lopez, Miao Liu, Michael Brandt, Stuart Reed
+(2013): Arctic Data Explorer SOLR Search software tools. The National Snow and
+Ice Data Center. Software. http://ezid.cdlib.org/id/doi:10.7265/N5JQ0XZM

data/bin/search_solr_tools

@@ -0,0 +1,87 @@
+#!/usr/bin/env ruby
+require 'search_solr_tools'
+require 'thor'
+
+class SolrHarvestCLI < Thor
+  desc 'harvest', 'Harvest from one of the ADE harvesters'
+  option :data_center, type: :array, required: true
+  option :environment, required: true
+  option :die_on_failure, type: :boolean
+
+  def harvest(die_on_failure = options[:die_on_failure] || false)
+    options[:data_center].each do |target|
+      puts target
+      begin
+        harvest_class = get_harvester_class(target)
+        harvester = harvest_class.new options[:environment], die_on_failure
+        harvester.harvest_and_delete
+      rescue => e
+        puts "harvest failed for #{target}: #{e.message}"
+        raise e if die_on_failure
+      end
+    end
+  end
+
+  desc 'list_harvesters', 'List all harvesters'
+  def list_harvesters
+    puts harvester_map.keys
+  end
+
+  desc 'delete_all', 'Delete all documents from the index'
+  option :environment, required: true
+  def delete_all
+    env = SearchSolrTools::SolrEnvironments[options[:environment]]
+    `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<delete><query>*:*</query></delete>'`
+    `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<commit/>'`
+  end
+
+  desc 'delete_all_auto_suggest', 'Delete all documents from the auto_suggest index'
+  option :environment, required: true
+  def delete_all_auto_suggest
+    env = SearchSolrTools::SolrEnvironments[options[:environment]]
+    `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<delete><query>*:*</query></delete>'`
+    `curl 'http://#{env[:host]}:#{env[:port]}/solr/update' -H 'Content-Type: text/xml; charset=utf-8' --data '<commit/>'`
+  end
+
+  desc 'delete_by_data_center', 'Force deletion of documents for a specific data center with timestamps before the passed timestamp in format iso8601 (2014-07-14T21:49:21Z)'
+  option :timestamp, required: true
+  option :environment, required: true
+  option :data_center,  required: true
+  def delete_by_data_center
+    harvester = get_harvester_class(options[:data_center]).new options[:environment]
+    harvester.delete_old_documents(options[:timestamp],
+                                   "data_centers:\"#{SearchSolrTools::Helpers::SolrFormat::DATA_CENTER_NAMES[options[:data_center].upcase.to_sym][:long_name]}\"",
+                                   SearchSolrTools::SolrEnvironments[harvester.environment][:collection_name],
+                                   true
+                                  )
+  end
+
+  no_tasks do
+    def harvester_map
+      {
+        'bco_dmo'            => SearchSolrTools::Harvesters::BcoDmo,
+        'cisl'               => SearchSolrTools::Harvesters::Cisl,
+        'echo'               => SearchSolrTools::Harvesters::Echo,
+        'eol'                => SearchSolrTools::Harvesters::Eol,
+        'ices'               => SearchSolrTools::Harvesters::Ices,
+        'nmi'                => SearchSolrTools::Harvesters::Nmi,
+        'nodc'               => SearchSolrTools::Harvesters::Nodc,
+        'rda'                => SearchSolrTools::Harvesters::Rda,
+        'usgs'               => SearchSolrTools::Harvesters::Usgs,
+        'tdar'               => SearchSolrTools::Harvesters::Tdar,
+        'pdc'                => SearchSolrTools::Harvesters::Pdc,
+        'nsidc'              => SearchSolrTools::Harvesters::NsidcJson,
+        'nsidc_auto_suggest' => SearchSolrTools::Harvesters::NsidcAutoSuggest,
+        'ade_auto_suggest'   => SearchSolrTools::Harvesters::AdeAutoSuggest
+      }
+    end
+
+    def get_harvester_class(data_center_name)
+      name = data_center_name.downcase.to_s
+      fail("Invalid data center #{name}") unless harvester_map.key?(name)
+
+      harvester_map[name]
+    end
+  end
+end
+SolrHarvestCLI.start(ARGV)

data/lib/search_solr_tools.rb

@@ -0,0 +1,8 @@
+require 'require_all'
+require_relative './search_solr_tools/config/environments'
+require_relative './search_solr_tools/version'
+
+require_rel './search_solr_tools/helpers'
+require_rel './search_solr_tools/selectors'
+require_rel './search_solr_tools/harvesters'
+require_rel './search_solr_tools/translators'

data/lib/search_solr_tools/config/environments.rb

@@ -0,0 +1,12 @@
+require 'yaml'
+
+module SearchSolrTools
+  # configuration to work with solr locally, or on integration/qa/staging/prod
+  module SolrEnvironments
+    YAML_ENVS = YAML.load_file(File.expand_path('../environments.yaml', __FILE__))
+
+    def self.[](env = :development)
+      YAML_ENVS[:common].merge(YAML_ENVS[env.to_sym])
+    end
+  end
+end

data/lib/search_solr_tools/config/environments.yaml

@@ -0,0 +1,73 @@
+:common:
+  :auto_suggest_collection_name: auto_suggest
+  :collection_name: nsidc_oai
+  :collection_path: solr
+  :port: 8983
+  :bcodmo_url: http://www.bco-dmo.org/nsidc/arctic-deployments.json
+  :cisl_url: https://www.aoncadis.org/oai/repository
+  :echo_url: https://api.echo.nasa.gov/catalog-rest/echo_catalog/datasets.echo10
+  :ices_url: http://geo.ices.dk/geonetwork/srv/en/csw
+  :nmi_url: http://access.met.no/metamod/oai
+  :nodc_url: http://data.nodc.noaa.gov/geoportal/csw
+  :pdc_url: http://www.polardata.ca/oai/provider
+  :rda_url: http://rda.ucar.edu/cgi-bin/oai
+  :tdar_url: http://core.tdar.org/search/rss
+  :usgs_url: https://www.sciencebase.gov/catalog/item/527cf4ede4b0850ea05182ee/csw
+  :eol:
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.SHEBA.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.SBI.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.PacMARS.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BASE.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ATLAS.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ARC_MIP.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.AMTS.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BOREAS.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BeringSea.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ARCSS.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BEST.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BSIERP.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.BARROW.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.DBO.thredds.xml
+    - http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ITEX.thredds.xml
+
+:local:
+  :host: localhost
+  :nsidc_dataset_metadata_url: http://integration.nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://integration.nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://liquid.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:dev:
+  host: dev.search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://integration.nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://integration.nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://liquid.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:integration:
+  :host: integration.search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://integration.nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://integration.nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://liquid.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:qa:
+  :host: qa.search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://qa.nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://qa.nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://brash.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:staging:
+  :host: staging.search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://staging.nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://staging.nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://freeze.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:blue:
+  :host: blue.search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://frozen.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso
+
+:production:
+  :host: search-solr.apps.int.nsidc.org
+  :nsidc_dataset_metadata_url: http://nsidc.org/api/dataset/metadata/
+  :nsidc_oai_identifiers_url: http://nsidc.org/api/dataset/metadata/oai?verb=ListIdentifiers&metadata_prefix=iso
+  :oai_url: http://frozen.colorado.edu:11580/api/dataset/2/oai?verb=ListRecords&metadata_prefix=iso

data/lib/search_solr_tools/harvesters/ade_auto_suggest.rb

@@ -0,0 +1,43 @@
+module SearchSolrTools
+  module Harvesters
+    class AdeAutoSuggest < AutoSuggest
+      def harvest_and_delete
+        puts 'Building auto-suggest indexes for ADE'
+        super(method(:harvest), "source:\"ADE\"", @env_settings[:auto_suggest_collection_name])
+      end
+
+      def harvest
+        url = "#{solr_url}/#{@env_settings[:collection_name]}/select?q=*%3A*&fq=source%3AADE&fq=spatial:[45.0,-180.0+TO+90.0,180.0]&rows=0&wt=json&indent=true&facet=true&facet.mincount=1&facet.sort=count&facet.limit=-1"
+        super url, fields
+      end
+
+      def fields
+        { 'full_keywords_and_parameters' => { weight: 2, source: 'ADE', creator: method(:keyword_creator) },
+          'full_authors' => { weight: 1, source: 'ADE', creator: method(:author_creator) }
+        }
+      end
+
+      def split_creator(value, count, field_weight, source, split_regex)
+        add_docs = []
+        value.downcase.split(split_regex).each do |v|
+          v = v.strip.chomp('/')
+          add_docs.concat(ade_length_limit_creator(v, count, field_weight, source)) unless v.nil? || v.empty?
+        end
+        add_docs
+      end
+
+      def keyword_creator(value, count, field_weight, source)
+        split_creator value, count, field_weight, source, %r{/ [\/ \>]+ /}
+      end
+
+      def author_creator(value, count, field_weight, source)
+        split_creator value, count, field_weight, source, %r{/;/}
+      end
+
+      def ade_length_limit_creator(value, count, field_weight, source)
+        return [] if value.length > 80
+        standard_add_creator value, count, field_weight, source
+      end
+    end
+  end
+end

data/lib/search_solr_tools/harvesters/auto_suggest.rb

@@ -0,0 +1,61 @@
+require 'json'
+require 'rest-client'
+
+module SearchSolrTools
+  module Harvesters
+    # Use the nsidc_oai core to populate the auto_suggest core
+    class AutoSuggest < Base
+      def initialize(env = 'development', die_on_failure = false)
+        super env, die_on_failure
+        @env_settings = SolrEnvironments[@environment] # super sets @environment.
+      end
+
+      private
+
+      def harvest(url, fields)
+        facet_response = fetch_auto_suggest_facet_data(url, fields)
+        add_docs = generate_add_hashes(facet_response, fields)
+        add_documents_to_solr(add_docs)
+      end
+
+      def standard_add_creator(value, count, field_weight, source)
+        count_weight = count <= 1 ? 0.4 : Math.log(count)
+        weight = field_weight * count_weight
+        [{ 'id' => "#{source}:#{value}", 'text_suggest' => value, 'source' => source, 'weight' => weight }]
+      end
+
+      def fetch_auto_suggest_facet_data(url, fields)
+        fields.each do |name, _config|
+          url += "&facet.field=#{name}"
+        end
+
+        serialized_facet_response = RestClient.get url
+        JSON.parse(serialized_facet_response)
+      end
+
+      def generate_add_hashes(facet_response, fields)
+        add_docs = []
+        facet_response['facet_counts']['facet_fields'].each do |facet_name, facet_values|
+          facet_values.each_slice(2) do |facet_value|
+            new_docs = fields[facet_name][:creator].call(facet_value[0], facet_value[1], fields[facet_name][:weight], fields[facet_name][:source])
+            add_docs.concat(new_docs)
+          end
+        end
+        add_docs
+      end
+
+      def add_documents_to_solr(add_docs)
+        if insert_solr_doc add_docs, Base::JSON_CONTENT_TYPE, @env_settings[:auto_suggest_collection_name]
+          puts "Added #{add_docs.size} auto suggest documents in one commit"
+        else
+          puts "Failed adding #{add_docs.size} documents in single commit, retrying one by one"
+          new_add_docs = []
+          add_docs.each do |doc|
+            new_add_docs << { 'add' => { 'doc' => doc } }
+          end
+          insert_solr_docs new_add_docs, Base::JSON_CONTENT_TYPE, @env_settings[:auto_suggest_collection_name]
+        end
+      end
+    end
+  end
+end