geo_combine 0.2.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: f0afe886cbbf89998cabcdad7c4daa67842f454d
-  data.tar.gz: bf8ea5de5455e02536043c038bcd96c0eae804c1
+SHA256:
+  metadata.gz: '0439618d4248a21c1f4efe91e863d01ad9b97f0798b7f855b60c15eab7457bbf'
+  data.tar.gz: a01ec4dc01b9c6d3dd39e34c178f2271b83d8c51ecf16226e8595a36ddbf37ea
 SHA512:
-  metadata.gz: 8e3155d7e08bd953c81f8ae501a691406e5ab606d66749a6cb7bd30262f7d21094a3db830af11c9ed65000d81cf88b0fcb5f8868b3f878f22f8d8ee701511dec
-  data.tar.gz: 90c54a422198f1c960550a19dcf993b8083e13cf8d477870485bb196535a90393f5bbd83ed8a94a810905b1b537838e671c4cd406e1aa014acf5a6603254226f
+  metadata.gz: 4c7c77f1cf1b60e7438f144d97a1af16126cdc1c9567ae2006cdcaa26c47fc0d5af8672e6368b5b139789b8f44f55cbadc1bd1f220b84b4a36c4bfc78dba3a65
+  data.tar.gz: d33c6c00ed6b39a6c2a8238ad9e14304bb59e450a031b8b8836635bb71d4a2deed3b8b0de68ca10c6e0ae9e9013f67acab42e765534475e2040af76f65a7a1ad

data/.travis.yml

@@ -2,6 +2,7 @@ sudo: false
 language: ruby
 cache: bundler
 rvm:
-  - 2.1.10
-  - 2.2.5
-  - 2.3.1
+  - 2.4.10
+  - 2.5.8
+  - 2.6.6
+  - 2.7.1

data/Gemfile

@@ -3,4 +3,5 @@ source 'https://rubygems.org'
 # Specify your gem's dependencies in geo_combine.gemspec
 gemspec
 
-gem 'coveralls', require: false
+gem 'coveralls', require: false
+gem 'byebug'

data/README.md

@@ -3,12 +3,13 @@
 [![Build Status](https://travis-ci.org/OpenGeoMetadata/GeoCombine.svg?branch=master)](https://travis-ci.org/OpenGeoMetadata/GeoCombine) | [![Coverage Status](https://coveralls.io/repos/OpenGeoMetadata/GeoCombine/badge.svg?branch=master)](https://coveralls.io/r/OpenGeoMetadata/GeoCombine?branch=master)
 
 
-
-A Ruby toolkit for managing geospatial metadata
+A Ruby toolkit for managing geospatial metadata, including:
+- tasks for cloning, updating, and indexing OpenGeoMetdata metadata
+- library for converting metadata between standards
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Add this line to your application's `Gemfile`:
 
 ```ruby
 gem 'geo_combine'
@@ -16,81 +17,161 @@ gem 'geo_combine'
 
 And then execute:
 
-    $ bundle
+    $ bundle install
 
 Or install it yourself as:
 
     $ gem install geo_combine
 
 ## Usage
-GeoCombine can be used as a set of rake tasks for cloning, updating, and indexing OpenGeoMetdata metdata. It can also be used as a Ruby library for converting metdata.
 
-### Transforming metadata
+### Converting metadata
 
 ```ruby
 # Create a new ISO19139 object
 > iso_metadata =  GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
 
-# Convert it to GeoBlacklight
+# Convert ISO to GeoBlacklight
 > iso_metadata.to_geoblacklight
 
 # Convert that to JSON
 > iso_metadata.to_geoblacklight.to_json
 
-# Convert ISO or FGDC to HTML
+# Convert ISO (or FGDC) to HTML
 > iso_metadata.to_html
 ```
 
-## Command line ##
+### OpenGeoMetadata
+
+#### Clone OpenGeoMetadata repositories locally
 
-GeoCombine's tasks can be run either as rake tasks or as standalone executables.
+```sh
+$ bundle exec rake geocombine:clone
+```
 
-### Clone all OpenGeoMetadata repositories
+Will clone all `edu.*`,` org.*`, and `uk.*` OpenGeoMetadata repositories into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
 
 ```sh
-$ rake geocombine:clone
+$ OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
 ```
 
+You can also specify a single repository:
+
 ```sh
-$ bundle exec geocombine clone
+$ bundle exec rake geocombine:clone[edu.stanford.purl]
 ```
 
-Will clone all edu.* OpenGeoMetadata repositories into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
+#### Update local OpenGeoMetadata repositories
 
 ```sh
-$ OGM_PATH='my/custom/location' rake geocombine:clone
+$ bundle exec rake geocombine:pull
 ```
 
-### Pull all OpenGeoMetadata repositories
+Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`).
+
+You can also specify a single repository:
 
 ```sh
-$ rake geocombine:pull
+$ bundle exec rake geocombine:pull[edu.stanford.purl]
 ```
 
+#### Index GeoBlacklight documents
+
+To index into Solr, GeoCombine requires a Solr instance that is running the
+[GeoBlacklight schema](https://github.com/geoblacklight/geoblacklight):
+
 ```sh
-$ bundle exec geocombine pull
+$ bundle exec rake geocombine:index
 ```
 
-Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`)
+Indexes the `geoblacklight.json` files in cloned repositories to a Solr index running at http://127.0.0.1:8983/solr
 
-### Index all of the GeoBlacklight documents
+##### Custom Solr location
+
+Solr location can also be specified by an environment variable `SOLR_URL`.
 
 ```sh
-$ rake geocombine:index
+$ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
 ```
 
+Depending on your Solr instance's performance characteristics, you may want to
+change the [`commitWithin` parameter](https://lucene.apache.org/solr/guide/6_6/updatehandlers-in-solrconfig.html) (in milliseconds):
+
 ```sh
-$ bundle exec geocombine index
+$ SOLR_COMMIT_WITHIN=100 bundle exec rake geocombine:index
 ```
 
-Indexes all of the `geoblacklight.json` files in cloned repositories to a Solr index running at http://127.0.0.1:8983/solr
+### Harvesting and indexing documents from GeoBlacklight sites
 
-#### Custom Solr location
+GeoCombine provides a Harvester class and rake task to harvest and index content from GeoBlacklight sites (or any site that follows the Blacklight API format). Given that the configurations can change from consumer to consumer and site to site, the class provides a relatively simple configuration API. This can be configured in an initializer, a wrapping rake task, or any other ruby context where the rake task our class would be invoked.
 
-Solr location can also be specified by an environment variable `SOLR_URL`.
+```sh
+bundle exec rake geocombine:geoblacklight_harvester:index[YOUR_CONFIGURED_SITE_KEY]
+```
+
+#### Harvester configuration
+
+Only the sites themselves are required to be configured but there are various configuration options that can (optionally) be supplied to modify the harvester's behavior.
+
+```ruby
+GeoCombine::GeoBlacklightHarvester.configure do
+  {
+    commit_within: '10000',
+    crawl_delay: 1, # All sites
+    debug: true,
+    SITE1: {
+      crawl_delay: 2, # SITE1 only
+      host: 'https://geoblacklight.example.edu',
+      params: {
+        f: {
+          dct_provenance_s: ['Institution']
+        }
+      }
+    },
+    SITE2: {
+      host: 'https://geoportal.example.edu',
+      params: {
+        q: '*'
+      }
+    }
+  }
+end
+```
+
+##### Crawl Delays (default: none)
+
+Crawl delays can be configured (in seconds) either globally for all sites or on a per-site basis. This will cause a delay for that number of seconds between each search results page (note that Blacklight 7 necessitates a lot of requests per results page and this only causes the delay per page of results)
+
+##### Solr's commitWithin (default: 5000 milliseconds)
+
+Solr's commitWithin option can be configured (in milliseconds) by passing a value under the commit_within key.
+
+##### Debugging (default: false)
+
+The harvester and indexer will only `puts` content when errors happen. It is possible to see some progress information by setting the debug configuration option.
+
+#### Transforming Documents
+
+You may need to transform documents that are harvested for various purposes (removing fields, adding fields, omitting a document all together, etc). You can configure some ruby code (a proc) that will take the document in, transform it, and return the transformed document. By default the indexer will remove the `score`, `timestamp`, and `_version_` fields from the documents harvested. If you provide your own transformer, you'll likely want to remove these fields in addition to the other transformations you provide.
+
+```ruby
+GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
+  # Removes "bogus_field" from the content we're harvesting
+  # in addition to some other solr fields we don't want
+  %w[_version_ score timestamp bogus_field].each do |field|
+    document.delete(field)
+  end
+
+  document
+end
+```
+
+## Tests
+
+To run the tests, use:
 
 ```sh
-$ SOLR_URL=http://www.example.com:1234/solr/collection rake geocombine:index
+$ bundle exec rake spec
 ```
 
 ## Contributing

data/geo_combine.gemspec

@@ -18,14 +18,16 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
+  spec.add_dependency 'activesupport'
   spec.add_dependency 'rsolr'
+  spec.add_dependency 'net-http-persistent', '~> 2.0' # pin since faraday (rsolr) doesn't work correctly with 3.x
   spec.add_dependency 'nokogiri'
   spec.add_dependency 'json-schema'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'
 
-  spec.add_development_dependency "bundler", "~> 1.7"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'rake'
   spec.add_development_dependency 'rspec'
   spec.add_development_dependency 'rspec-html-matchers'
 end

data/lib/geo_combine.rb

@@ -20,7 +20,7 @@ module GeoCombine
     ##
     # Creates a new GeoCombine::Metadata object, where metadata parameter is can
     # be a File path or String of XML
-    # @param [String] metadata can be a File path 
+    # @param [String] metadata can be a File path
     # "./tmp/edu.stanford.purl/bb/338/jh/0716/iso19139.xml" or a String of XML
     # metadata
     def initialize metadata
@@ -58,12 +58,19 @@ require 'geo_combine/geometry_types'
 
 # Require helper mixins
 require 'geo_combine/formatting'
+require 'geo_combine/bounding_box'
 
 # Require additional classes
 require 'geo_combine/fgdc'
 require 'geo_combine/geoblacklight'
 require 'geo_combine/iso19139'
 require 'geo_combine/esri_open_data'
+require 'geo_combine/ckan_metadata'
+require 'geo_combine/ogp'
+
+# Require harvesting/indexing files
+require 'geo_combine/geo_blacklight_harvester'
 
 # Require gem files
 require 'geo_combine/version'
+require 'geo_combine/railtie' if defined?(Rails)

data/lib/geo_combine/bounding_box.rb

@@ -0,0 +1,71 @@
+module GeoCombine
+  class BoundingBox
+    attr_reader :west, :south, :east, :north
+
+    ##
+    # @param [String, Integer, Float] west
+    # @param [String, Integer, Float] south
+    # @param [String, Integer, Float] east
+    # @param [String, Integer, Float] north
+    def initialize(west:, south:, east:, north:)
+      @west = west.to_f
+      @south = south.to_f
+      @east = east.to_f
+      @north = north.to_f
+    end
+
+    ##
+    # Returns a bounding box in ENVELOPE syntax
+    # @return [String]
+    def to_envelope
+      "ENVELOPE(#{west}, #{east}, #{north}, #{south})"
+    end
+
+    def valid?
+      [south, north].map do |coord|
+        next if (-90..90).cover?(coord)
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "#{coord} should be in range -90 90"
+      end
+      [east, west].map do |coord|
+        next if (-180..180).cover?(coord)
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "#{coord} should be in range -180 180"
+      end
+      if west > east
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "east #{east} should be greater than or equal to west #{west}"
+      end
+      if south > north
+        raise GeoCombine::Exceptions::InvalidGeometry,
+              "north #{north} should be greater than or equal to south #{south}"
+      end
+      true
+    end
+
+    def self.from_envelope(envelope)
+      return if envelope.nil?
+      envelope = envelope[/.*ENVELOPE\(([^\)]*)/, 1].split(',')
+      new(
+        west: envelope[0],
+        south: envelope[3],
+        east: envelope[1],
+        north: envelope[2]
+      )
+    end
+
+    ##
+    # @param [String] spatial w,s,e,n or w s e n
+    # @param [String] delimiter "," or " "
+    def self.from_string_delimiter(spatial, delimiter: ',')
+      return if spatial.nil?
+      spatial = spatial.split(delimiter)
+      new(
+        west: spatial[0],
+        south: spatial[1],
+        east: spatial[2],
+        north: spatial[3]
+      )
+    end
+  end
+end

data/lib/geo_combine/ckan_metadata.rb

@@ -0,0 +1,112 @@
+module GeoCombine
+  class CkanMetadata
+    MAX_STRING_LENGTH = 32765 # Solr limit
+
+    attr_reader :metadata
+    def initialize(metadata)
+      @metadata = JSON.parse(metadata)
+    end
+
+    ##
+    # Creates and returns a Geoblacklight schema object from this metadata
+    # @return [GeoCombine::Geoblacklight]
+    def to_geoblacklight
+      GeoCombine::Geoblacklight.new(geoblacklight_terms.to_json)
+    end
+
+    ##
+    # Builds a Geoblacklight Schema type hash from Esri Open Data portal
+    # metadata
+    # @return [Hash]
+    def geoblacklight_terms
+      {
+        dc_identifier_s: @metadata['id'],
+        dc_title_s: @metadata['title'],
+        dc_rights_s: 'Public',
+        layer_geom_type_s: 'Not Specified',
+        dct_provenance_s: organization['title'],
+        dc_description_s: @metadata['notes'].respond_to?(:[]) ? @metadata['notes'][0..MAX_STRING_LENGTH] : nil,
+        layer_slug_s: @metadata['name'],
+        solr_geom: envelope,
+        dc_subject_sm: subjects,
+        dct_references_s: external_references.to_json.to_s,
+        dc_format_s: downloadable? ? 'ZIP' : nil # TODO: we only allow direct ZIP file downloads
+      }.select { |_k, v| !v.nil? }
+    end
+
+    def organization
+      @metadata['organization'] || { title: '' }
+    end
+
+    def envelope
+      return envelope_from_bbox unless envelope_from_bbox.nil?
+      return envelope_from_spatial(',') unless envelope_from_spatial(',').nil?
+      return envelope_from_spatial(' ') unless envelope_from_spatial(' ').nil?
+    end
+
+    def envelope_from_bbox
+      bbox = GeoCombine::BoundingBox.new(
+        west: extras('bbox-west-long'),
+        south: extras('bbox-south-lat'),
+        east: extras('bbox-east-long'),
+        north: extras('bbox-north-lat')
+      )
+      begin
+        return bbox.to_envelope if bbox.valid?
+      rescue GeoCombine::Exceptions::InvalidGeometry
+        return nil
+      end
+    end
+
+    def envelope_from_spatial(delimiter)
+      bbox = GeoCombine::BoundingBox.from_string_delimiter(
+        extras('spatial'),
+        delimiter: delimiter
+      )
+      begin
+        return bbox.to_envelope if bbox.valid?
+      rescue GeoCombine::Exceptions::InvalidGeometry
+        return nil
+      end
+    end
+
+    def subjects
+      extras('tags').split(',').map(&:strip)
+    end
+
+    def extras(key)
+      if @metadata['extras']
+        @metadata['extras'].select { |h| h['key'] == key }.collect { |v| v['value'] }[0] || ''
+      else
+        ''
+      end
+    end
+
+    def external_references
+      h = {
+        'http://schema.org/url' => resource_urls('information').first
+      }
+
+      if downloadable?
+        h['http://schema.org/downloadUrl'] = resource_urls('download').first
+      end
+
+      h.select { |_k, v| !v.nil? }
+    end
+
+    def downloadable?
+      resource_urls('download').first =~ /.*\.zip/im
+    end
+
+    def resources(type)
+      return [] if @metadata['resources'].nil?
+      @metadata['resources'].select { |resource| resource['resource_locator_function'] == type }
+    end
+
+    def resource_urls(type)
+      resources(type).collect do |resource|
+        resource['url'] if resource.respond_to?(:[])
+      end
+    end
+  end
+end