geo_combine 0.1.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 8627401efabab03d06f210ee160e7e09c7223398
-  data.tar.gz: dcdc93c201e689dc2c9b8245ac2076617af34e39
+SHA256:
+  metadata.gz: a403eec66c0f81cd06bed4033f411ed137276277e6f9a6eb0a672c64428dcca0
+  data.tar.gz: 7c6e8a61d3d8783a633edac40c57c435a4b6b3d9bd17ff1ae272e794fb6c2b56
 SHA512:
-  metadata.gz: 9c957f3b783684f18c18c3265349c7efa4db3aa12094f573e9d4d9e5a2366f3464e390add7a12734213ca1801c5ef3d0d820a8139fdce47c07ba99e037f71e5f
-  data.tar.gz: e6bedee395328db1deed473721231e7a5d678e0bfa2a9b038e3f7de596e99622cc7dff7b4dc2d6db14d2d5734339d01fa9d5a462bfbe4363872afe87c98ecb48
+  metadata.gz: b995db35d3452c377d93a9384018e54c01730b76462e5109d3a4ab3e7e24ddcbf3a8fa1b7703b2e571832a054d8a43459a34b18df9b5af418e87107c8509a873
+  data.tar.gz: fc129701c6e2fb138e9f43aab3b2b6e84dc68696813cbb1fb5c7ede89f364c88b8d5717a3e477fc030dfd2ee588a335df0274bf4005f1434c14a841d0f92e0de

data/.travis.yml

@@ -1,3 +1,8 @@
+sudo: false
 language: ruby
+cache: bundler
 rvm:
-  - 2.1.5
+  - 2.4.10
+  - 2.5.8
+  - 2.6.6
+  - 2.7.1

data/Gemfile

@@ -3,4 +3,5 @@ source 'https://rubygems.org'
 # Specify your gem's dependencies in geo_combine.gemspec
 gemspec
 
-gem 'coveralls', require: false
+gem 'coveralls', require: false
+gem 'byebug'

data/README.md

@@ -3,12 +3,13 @@
 [![Build Status](https://travis-ci.org/OpenGeoMetadata/GeoCombine.svg?branch=master)](https://travis-ci.org/OpenGeoMetadata/GeoCombine) | [![Coverage Status](https://coveralls.io/repos/OpenGeoMetadata/GeoCombine/badge.svg?branch=master)](https://coveralls.io/r/OpenGeoMetadata/GeoCombine?branch=master)
 
 
-
-A Ruby toolkit for managing geospatial metadata
+A Ruby toolkit for managing geospatial metadata, including:
+- tasks for cloning, updating, and indexing OpenGeoMetdata metadata
+- library for converting metadata between standards
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Add this line to your application's `Gemfile`:
 
 ```ruby
 gem 'geo_combine'
@@ -16,74 +17,158 @@ gem 'geo_combine'
 
 And then execute:
 
-    $ bundle
+    $ bundle install
 
 Or install it yourself as:
 
     $ gem install geo_combine
 
 ## Usage
-GeoCombine can be used as a set of rake tasks for cloning, updating, and indexing OpenGeoMetdata metdata. It can also be used as a Ruby library for converting metdata.
 
-### Transforming metadata
+### Converting metadata
 
 ```ruby
 # Create a new ISO19139 object
 > iso_metadata =  GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
 
-# Convert it to GeoBlacklight
+# Convert ISO to GeoBlacklight
 > iso_metadata.to_geoblacklight
 
 # Convert that to JSON
 > iso_metadata.to_geoblacklight.to_json
 
-# Convert ISO or FGDC to HTML
+# Convert ISO (or FGDC) to HTML
 > iso_metadata.to_html
 ```
 
-## Command line ##
+### OpenGeoMetadata
+
+#### Clone OpenGeoMetadata repositories locally
 
-GeoCombine's tasks can be run either as rake tasks or as standalone executables.
+```sh
+$ bundle exec rake geocombine:clone
+```
 
-### Clone all OpenGeoMetadata repositories
+Will clone all `edu.*`,` org.*`, and `uk.*` OpenGeoMetadata repositories into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
 
 ```sh
-$ rake geocombine:clone
+$ OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
 ```
 
+You can also specify a single repository:
+
 ```sh
-$ bundle exec geocombine clone
+$ bundle exec rake geocombine:clone[edu.stanford.purl]
 ```
 
-Will clone all edu.* OpenGeoMetadata repositories into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
+#### Update local OpenGeoMetadata repositories
 
 ```sh
-$ OGM_PATH='my/custom/location' rake geocombine:clone
+$ bundle exec rake geocombine:pull
 ```
 
-### Pull all OpenGeoMetadata repositories
+Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`).
+
+You can also specify a single repository:
 
 ```sh
-$ rake geocombine:pull
+$ bundle exec rake geocombine:pull[edu.stanford.purl]
 ```
 
+#### Index GeoBlacklight documents
+
+To index into Solr, GeoCombine requires a Solr instance that is running the
+[GeoBlacklight schema](https://github.com/geoblacklight/geoblacklight):
+
 ```sh
-$ bundle exec geocombine pull
+$ bundle exec rake geocombine:index
 ```
 
-Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`)
+Indexes the `geoblacklight.json` files in cloned repositories to a Solr index running at http://127.0.0.1:8983/solr
 
-### Index all of the GeoBlacklight documents
+##### Custom Solr location
+
+Solr location can also be specified by an environment variable `SOLR_URL`.
 
 ```sh
-$ rake geocombine:index
+$ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
 ```
 
+Depending on your Solr instance's performance characteristics, you may want to
+change the [`commitWithin` parameter](https://lucene.apache.org/solr/guide/6_6/updatehandlers-in-solrconfig.html) (in milliseconds):
+
 ```sh
-$ bundle exec geocombine index
+$ SOLR_COMMIT_WITHIN=100 bundle exec rake geocombine:index
+```
+
+### Harvesting and indexing documents from GeoBlacklight sites
+
+GeoCombine provides a Harvester class and rake task to harvest and index content from GeoBlacklight sites (or any site that follows the Blacklight API format). Given that the configurations can change from consumer to consumer and site to site, the class provides a relatively simple configuration API. This can be configured in an initializer, a wrapping rake task, or any other ruby context where the rake task our class would be invoked.
+
+#### Harvester configuration
+
+Only the sites themselves are required to be configured but there are various configuration options that can (optionally) be supplied to modify the harvester's behavior.
+
+```ruby
+GeoCombine::GeoBlacklightIndexer.configure do
+  {
+    commit_within: '10000',
+    crawl_delay: 1, # All sites
+    debug: true,
+    SITE1: {
+      crawl_delay: 2, # SITE1 only
+      host: 'https://geoblacklight.example.edu',
+      params: {
+        f: {
+          dct_provenance_s: ['Institution']
+        }
+      }
+    },
+    SITE2: {
+      host: 'https://geoportal.example.edu',
+      params: {
+        q: '*'
+      }
+    }
+  }
+end
+```
+
+##### Crawl Delays (default: none)
+
+Crawl delays can be configured (in seconds) either globally for all sites or on a per-site basis. This will cause a delay for that number of seconds between each search results page (note that Blacklight 7 necessitates a lot of requests per results page and this only causes the delay per page of results)
+
+##### Solr's commitWithin (default: 5000 milliseconds)
+
+Solr's commitWithin option can be configured (in milliseconds) by passing a value under the commit_within key.
+
+##### Debugging (default: false)
+
+The harvester and indexer will only `puts` content when errors happen. It is possible to see some progress information by setting the debug configuration option.
+
+#### Transforming Documents
+
+You may need to transform documents that are harvested for various purposes (removing fields, adding fields, omitting a document all together, etc). You can configure some ruby code (a proc) that will take the document in, transform it, and return the transformed document. By default the indexer will remove the `score`, `timestamp`, and `_version_` fields from the documents harvested. If you provide your own transformer, you'll likely want to remove these fields in addition to the other transformations you provide.
+
+```ruby
+GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
+  # Removes "bogus_field" from the content we're harvesting
+  # in addition to some other solr fields we don't want
+  %w[_version_ score timestamp bogus_field].each do |field|
+    document.delete(field)
+  end
+
+  document
+end
 ```
 
-Indexs all of the `geoblacklight.json` files in cloned repositories to a Solr index running at http://127.0.0.1:8983/solr
+## Tests
+
+To run the tests, use:
+
+```sh
+$ bundle exec rake spec
+```
 
 ## Contributing
 

data/geo_combine.gemspec

@@ -18,14 +18,16 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
+  spec.add_dependency 'activesupport'
   spec.add_dependency 'rsolr'
+  spec.add_dependency 'net-http-persistent', '~> 2.0' # pin since faraday (rsolr) doesn't work correctly with 3.x
   spec.add_dependency 'nokogiri'
   spec.add_dependency 'json-schema'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'
 
-  spec.add_development_dependency "bundler", "~> 1.7"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'rake'
   spec.add_development_dependency 'rspec'
   spec.add_development_dependency 'rspec-html-matchers'
 end

data/lib/geo_combine.rb

@@ -20,7 +20,7 @@ module GeoCombine
     ##
     # Creates a new GeoCombine::Metadata object, where metadata parameter is can
     # be a File path or String of XML
-    # @param [String] metadata can be a File path 
+    # @param [String] metadata can be a File path
     # "./tmp/edu.stanford.purl/bb/338/jh/0716/iso19139.xml" or a String of XML
     # metadata
     def initialize metadata
@@ -48,6 +48,9 @@ module GeoCombine
   end
 end
 
+# Require custom exceptions
+require 'geo_combine/exceptions'
+
 # Require translation mixins
 require 'geo_combine/formats'
 require 'geo_combine/subjects'
@@ -55,12 +58,19 @@ require 'geo_combine/geometry_types'
 
 # Require helper mixins
 require 'geo_combine/formatting'
+require 'geo_combine/bounding_box'
 
 # Require additional classes
 require 'geo_combine/fgdc'
 require 'geo_combine/geoblacklight'
 require 'geo_combine/iso19139'
 require 'geo_combine/esri_open_data'
+require 'geo_combine/ckan_metadata'
+require 'geo_combine/ogp'
+
+# Require harvesting/indexing files
+require 'geo_combine/geo_blacklight_harvester'
 
 # Require gem files
 require 'geo_combine/version'
+require 'geo_combine/railtie' if defined?(Rails)

data/lib/geo_combine/bounding_box.rb

@@ -0,0 +1,71 @@
+module GeoCombine
+  class BoundingBox
+    attr_reader :west, :south, :east, :north
+
+    ##
+    # @param [String, Integer, Float] west
+    # @param [String, Integer, Float] south
+    # @param [String, Integer, Float] east
+    # @param [String, Integer, Float] north
+    def initialize(west:, south:, east:, north:)
+      @west = west.to_f
+      @south = south.to_f
+      @east = east.to_f
+      @north = north.to_f
+    end
+
+    ##
+    # Returns a bounding box in ENVELOPE syntax
+    # @return [String]
+    def to_envelope
+      "ENVELOPE(#{west}, #{east}, #{north}, #{south})"
+    end
+
+    def valid?
+      [south, north].map do |coord|
+        next if (-90..90).cover?(coord)
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "#{coord} should be in range -90 90"
+      end
+      [east, west].map do |coord|
+        next if (-180..180).cover?(coord)
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "#{coord} should be in range -180 180"
+      end
+      if west > east
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "east #{east} should be greater than or equal to west #{west}"
+      end
+      if south > north
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "north #{north} should be greater than or equal to south #{south}"
+      end
+      true
+    end
+
+    def self.from_envelope(envelope)
+      return if envelope.nil?
+      envelope = envelope[/.*ENVELOPE\(([^\)]*)/, 1].split(',')
+      new(
+        west: envelope[0],
+        south: envelope[3],
+        east: envelope[1],
+        north: envelope[2]
+      )
+    end
+
+    ##
+    # @param [String] spatial w,s,e,n or w s e n
+    # @param [String] delimiter "," or " "
+    def self.from_string_delimiter(spatial, delimiter: ',')
+      return if spatial.nil?
+      spatial = spatial.split(delimiter)
+      new(
+        west: spatial[0],
+        south: spatial[1],
+        east: spatial[2],
+        north: spatial[3]
+      )
+    end
+  end
+end

data/lib/geo_combine/ckan_metadata.rb

@@ -0,0 +1,112 @@
+module GeoCombine
+  class CkanMetadata
+    MAX_STRING_LENGTH = 32765 # Solr limit
+
+    attr_reader :metadata
+    def initialize(metadata)
+      @metadata = JSON.parse(metadata)
+    end
+
+    ##
+    # Creates and returns a Geoblacklight schema object from this metadata
+    # @return [GeoCombine::Geoblacklight]
+    def to_geoblacklight
+      GeoCombine::Geoblacklight.new(geoblacklight_terms.to_json)
+    end
+
+    ##
+    # Builds a Geoblacklight Schema type hash from Esri Open Data portal
+    # metadata
+    # @return [Hash]
+    def geoblacklight_terms
+      {
+        dc_identifier_s: @metadata['id'],
+        dc_title_s: @metadata['title'],
+        dc_rights_s: 'Public',
+        layer_geom_type_s: 'Not Specified',
+        dct_provenance_s: organization['title'],
+        dc_description_s: @metadata['notes'].respond_to?(:[]) ? @metadata['notes'][0..MAX_STRING_LENGTH] : nil,
+        layer_slug_s: @metadata['name'],
+        solr_geom: envelope,
+        dc_subject_sm: subjects,
+        dct_references_s: external_references.to_json.to_s,
+        dc_format_s: downloadable? ? 'ZIP' : nil # TODO: we only allow direct ZIP file downloads
+      }.select { |_k, v| !v.nil? }
+    end
+
+    def organization
+      @metadata['organization'] || { title: '' }
+    end
+
+    def envelope
+      return envelope_from_bbox unless envelope_from_bbox.nil?
+      return envelope_from_spatial(',') unless envelope_from_spatial(',').nil?
+      return envelope_from_spatial(' ') unless envelope_from_spatial(' ').nil?
+    end
+
+    def envelope_from_bbox
+      bbox = GeoCombine::BoundingBox.new(
+        west: extras('bbox-west-long'),
+        south: extras('bbox-south-lat'),
+        east: extras('bbox-east-long'),
+        north: extras('bbox-north-lat')
+      )
+      begin
+        return bbox.to_envelope if bbox.valid?
+      rescue GeoCombine::Exceptions::InvalidGeometry
+        return nil
+      end
+    end
+
+    def envelope_from_spatial(delimiter)
+      bbox = GeoCombine::BoundingBox.from_string_delimiter(
+        extras('spatial'),
+        delimiter: delimiter
+      )
+      begin
+        return bbox.to_envelope if bbox.valid?
+      rescue GeoCombine::Exceptions::InvalidGeometry
+        return nil
+      end
+    end
+
+    def subjects
+      extras('tags').split(',').map(&:strip)
+    end
+
+    def extras(key)
+      if @metadata['extras']
+        @metadata['extras'].select { |h| h['key'] == key }.collect { |v| v['value'] }[0] || ''
+      else
+        ''
+      end
+    end
+
+    def external_references
+      h = {
+        'http://schema.org/url' => resource_urls('information').first
+      }
+
+      if downloadable?
+        h['http://schema.org/downloadUrl'] = resource_urls('download').first
+      end
+
+      h.select { |_k, v| !v.nil? }
+    end
+
+    def downloadable?
+      resource_urls('download').first =~ /.*\.zip/im
+    end
+
+    def resources(type)
+      return [] if @metadata['resources'].nil?
+      @metadata['resources'].select { |resource| resource['resource_locator_function'] == type }
+    end
+
+    def resource_urls(type)
+      resources(type).collect do |resource|
+        resource['url'] if resource.respond_to?(:[])
+      end
+    end
+  end
+end