geo_combine 0.10.0 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: cf25387203a718f8ae41365d87fae7d1868ea1416bf8ae9a33d11b6fef21171d
4
- data.tar.gz: 7cbdec148a45206e21090b123390ac129d705c502e2ed1bd2c0d723b81afd51c
3
+ metadata.gz: 21614ec82bcbe5c5a7bb1291e398795f1baff048552746335f18f315da677c9e
4
+ data.tar.gz: 6c94a18a0f43fd141f5967930a909efac60ab560ac91a0759e48cc45fdab39b5
5
5
  SHA512:
6
- metadata.gz: 1319f17633d5ee99fe3706dcdae9b7a327339b658ee8677b678aa23132b8b48b031589b217399c53078625b5a3f19cbdd11f13a00313edbe563a0ea8b04c651b
7
- data.tar.gz: 521143da86eabc8ec0d89a45c82d8ccf58f2ab6bab03bf06357fd124e3601898a2ee8cd9a522b5d7a126d0e106e93f4790e84fd915186439eec7a7956814cf7c
6
+ metadata.gz: 75ce7b76dfa6ad5905410e6b38a18ee98185c50d5d116c8fc5794eba3376e85a05d2e2ad7d70d6bde8de7b0073df22f15cb28759633e0d3c6e546e24a13dee82
7
+ data.tar.gz: cbd5ccc9eac444e599e0dce59fda1f23ef752cec71508a83fbe32ea1b2785b5a8c52d66c4ebcebc81a913aaccf6f677f5555622a43a8e16e902ab5f3379cbfa0
data/README.md CHANGED
@@ -1,16 +1,20 @@
1
1
  # GeoCombine
2
2
 
3
3
  ![CI](https://github.com/OpenGeoMetadata/GeoCombine/actions/workflows/ruby.yml/badge.svg)
4
- | [![Coverage Status](https://img.shields.io/badge/coverage-95%25-brightgreen)]()
5
- | [![Gem Version](https://img.shields.io/gem/v/geo_combine.svg)](https://github.com/OpenGeoMetadata/GeoCombine/releases)
4
+ [![Coverage Status](https://img.shields.io/badge/coverage-95%25-brightgreen)]()
5
+ [![Gem Version](https://img.shields.io/gem/v/geo_combine.svg)](https://github.com/OpenGeoMetadata/GeoCombine/releases)
6
6
 
7
7
  A Ruby toolkit for managing geospatial metadata, including:
8
8
 
9
- - tasks for cloning, updating, and indexing OpenGeoMetadata metadata
10
- - library for converting metadata between standards
9
+ - harvesting and indexing records from OpenGeoMetadata
10
+ - harvesting records directly from a GeoBlacklight instance
11
+ - converting geospatial metadata records between standards
11
12
 
12
13
  ## Installation
13
14
 
15
+ > [!TIP]
16
+ > If you're using GeoBlacklight, GeoCombine is already bundled with your installation – you can skip this section.
17
+
14
18
  Add this line to your application's `Gemfile`:
15
19
 
16
20
  ```ruby
@@ -20,127 +24,64 @@ gem 'geo_combine'
20
24
  And then execute:
21
25
 
22
26
  ```sh
23
- $ bundle install
27
+ bundle install
24
28
  ```
25
29
 
26
30
  Or install it yourself as:
27
31
 
28
32
  ```sh
29
- $ gem install geo_combine
33
+ gem install geo_combine
30
34
  ```
31
35
 
32
36
  ## Usage
33
37
 
34
- ### Converting metadata
35
-
36
- #### Converting metadata into GeoBlacklight JSON
37
-
38
- GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
39
-
40
- ```ruby
41
- GeoCombine::Iso19139 # ISO 19139 XML
42
- GeoCombine::OGP # OpenGeoPortal JSON
43
- GeoCombine::Fgdc # FGDC XML
44
- GeoCombine::EsriOpenData # Esri Open Data Portal JSON
45
- GeoCombine::CkanMetadata # CKAN JSON
46
- ```
47
-
48
- An example for converting an ISO 19139 XML record:
49
-
50
- ```ruby
51
- # Create a new ISO19139 object
52
- > iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
53
-
54
- # Convert to GeoBlacklight's metadata format
55
- > iso_metadata.to_geoblacklight
56
-
57
- # Output it as JSON instead of a Ruby hash
58
- > iso_metadata.to_geoblacklight.to_json
59
- ```
60
-
61
- Some formats also support conversion into HTML for display in a web browser:
62
-
63
- ```ruby
64
- # Create a new ISO19139 object
65
- > iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
66
-
67
- # Convert ISO to HTML
68
- > iso_metadata.to_html
69
- ```
70
-
71
- ### Migrating metadata
72
-
73
- You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
74
-
75
- Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
76
-
77
- ```ruby
78
- # Load a record in geoblacklight v1 schema
79
- record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
80
-
81
- # Migrate it to Aardvark schema
82
- GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
83
- ```
84
-
85
- Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
86
-
87
- ```ruby
88
- # You can store this mapping as a JSON or CSV file and load it into a hash
89
- id_map = {
90
- 'My Collection 1' => 'institution:my-collection-1',
91
- 'My Collection 2' => 'institution:my-collection-2'
92
- }
93
-
94
- GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
95
- ```
96
-
97
38
  ### Downloading metadata from OpenGeoMetadata
98
39
 
99
- #### Logging
40
+ #### Schema version
100
41
 
101
- Some of the tools and scripts in this gem use Ruby's `Logger` class to print information to `$stderr`. By default, the log level is set to `Logger::INFO`. For more verbose information, you can set the `LOG_LEVEL` environment variable to `DEBUG`:
42
+ By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the `SCHEMA_VERSION` environment variable:
102
43
 
103
44
  ```sh
104
- $ export LOG_LEVEL=DEBUG
45
+ # Only fetch and index schema version 1.0 records
46
+ export SCHEMA_VERSION=1.0
105
47
  ```
106
48
 
107
- #### Schema version
49
+ #### Restricted records
108
50
 
109
- By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the `SCHEMA_VERSION` environment variable:
51
+ By default, GeoCombine will only fetch and index records that are not marked as restricted. If you want to include restricted records, you can set the `OGM_SKIP_RESTRICTED` environment variable to `false`:
110
52
 
111
53
  ```sh
112
- # Only fetch and index schema version 1.0 records
113
- $ export SCHEMA_VERSION=1.0
54
+ export OGM_SKIP_RESTRICTED=false
114
55
  ```
115
56
 
116
57
  #### Clone OpenGeoMetadata repositories locally
117
58
 
118
59
  ```sh
119
- $ bundle exec rake geocombine:clone
60
+ bundle exec rake geocombine:clone
120
61
  ```
121
62
 
122
63
  Will clone all OpenGeoMetadata repositories containing metadata matching the `SCHEMA_VERSION` into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
123
64
 
124
65
  ```sh
125
- $ OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
66
+ OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
126
67
  ```
127
68
 
128
69
  You can also specify a single repository:
129
70
 
130
71
  ```sh
131
- $ bundle exec rake geocombine:clone[edu.stanford.purl]
72
+ bundle exec rake geocombine:clone[edu.stanford.purl]
132
73
  ```
133
74
 
134
75
  _Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
135
76
 
136
77
  ```sh
137
- $ bundle exec rake geocombine:clone\[edu.stanford.purl\]
78
+ bundle exec rake geocombine:clone\[edu.stanford.purl\]
138
79
  ```
139
80
 
140
81
  #### Update local OpenGeoMetadata repositories
141
82
 
142
83
  ```sh
143
- $ bundle exec rake geocombine:pull
84
+ bundle exec rake geocombine:pull
144
85
  ```
145
86
 
146
87
  Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`).
@@ -148,22 +89,25 @@ Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometada
148
89
  You can also specify a single repository:
149
90
 
150
91
  ```sh
151
- $ bundle exec rake geocombine:pull[edu.stanford.purl]
92
+ bundle exec rake geocombine:pull[edu.stanford.purl]
152
93
  ```
153
94
 
154
95
  _Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
155
96
 
156
97
  ```sh
157
- $ bundle exec rake geocombine:pull\[edu.stanford.purl\]
98
+ bundle exec rake geocombine:pull\[edu.stanford.purl\]
158
99
  ```
159
100
 
160
101
  #### Index GeoBlacklight documents
161
102
 
103
+ > [!TIP]
104
+ > If you're using GeoBlacklight, GeoCombine will automatically detect the path to your Solr index if you run it in the directory where GeoBlacklight is installed.
105
+
162
106
  To index into Solr, GeoCombine requires a Solr instance that is running the
163
107
  [GeoBlacklight schema](https://github.com/geoblacklight/geoblacklight):
164
108
 
165
109
  ```sh
166
- $ bundle exec rake geocombine:index
110
+ bundle exec rake geocombine:index
167
111
  ```
168
112
 
169
113
  If Blacklight is installed in the ruby environment and a solr index is configured, the rake task will use the solr index configured in the Blacklight application (this is the case when invoking GeoCombine from your GeoBlacklight installation). If Blacklight is unavailable, the rake task will try to find a Solr instance running at `http://localhost:8983/solr/blacklight-core`.
@@ -171,24 +115,40 @@ If Blacklight is installed in the ruby environment and a solr index is configure
171
115
  You can also set the Solr instance URL using `SOLR_URL`:
172
116
 
173
117
  ```sh
174
- $ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
118
+ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
119
+ ```
120
+
121
+ #### Logging
122
+
123
+ Some of the tools and scripts in this gem use Ruby's `Logger` class to print information to `$stderr`. By default, the log level is set to `Logger::INFO`. For more verbose information, you can set the `LOG_LEVEL` environment variable to `DEBUG`:
124
+
125
+ ```sh
126
+ export LOG_LEVEL=DEBUG
175
127
  ```
176
128
 
177
129
  ### Indexing local documents
178
130
 
179
- To index an arbitrary collection of records in a custom directory, run one of the following:
131
+ To index a collection of records in a local directory, you can pass the directory as a `rake` argument:
132
+
133
+ ```sh
134
+ bundle exec rake geocombine:index\[/path/to/your/files\]
135
+ ```
136
+
137
+ Or you can provide it as `OGM_PATH`:
180
138
 
181
139
  ```sh
182
- rake geocombine:index\[/path/to/your/files\]
183
140
  OGM_PATH=/path/to/your/files rake geocombine:index
184
141
  ```
185
142
 
186
- ### Indexing a single document
143
+ For a single file, it works the same way:
187
144
 
188
- To index a single arbitrary record, run one of the following:
145
+ ```sh
146
+ bundle exec rake geocombine:index\[/path/to/your/file.json\]
147
+ ```
148
+
149
+ Or:
189
150
 
190
151
  ```sh
191
- rake geocombine:index\[/path/to/your/file.json\]
192
152
  OGM_PATH=/path/to/your/file.json rake geocombine:index
193
153
  ```
194
154
 
@@ -253,12 +213,75 @@ GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
253
213
  end
254
214
  ```
255
215
 
216
+ ### Converting metadata
217
+
218
+ #### Converting metadata into GeoBlacklight JSON
219
+
220
+ GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
221
+
222
+ ```ruby
223
+ GeoCombine::Iso19139 # ISO 19139 XML
224
+ GeoCombine::OGP # OpenGeoPortal JSON
225
+ GeoCombine::Fgdc # FGDC XML
226
+ GeoCombine::EsriOpenData # Esri Open Data Portal JSON
227
+ GeoCombine::CkanMetadata # CKAN JSON
228
+ ```
229
+
230
+ An example for converting an ISO 19139 XML record:
231
+
232
+ ```ruby
233
+ # Create a new ISO19139 object
234
+ > iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
235
+
236
+ # Convert to GeoBlacklight's metadata format
237
+ > iso_metadata.to_geoblacklight
238
+
239
+ # Output it as JSON instead of a Ruby hash
240
+ > iso_metadata.to_geoblacklight.to_json
241
+ ```
242
+
243
+ Some formats also support conversion into HTML for display in a web browser:
244
+
245
+ ```ruby
246
+ # Create a new ISO19139 object
247
+ > iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
248
+
249
+ # Convert ISO to HTML
250
+ > iso_metadata.to_html
251
+ ```
252
+
253
+ ### Migrating metadata
254
+
255
+ You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
256
+
257
+ Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
258
+
259
+ ```ruby
260
+ # Load a record in geoblacklight v1 schema
261
+ record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
262
+
263
+ # Migrate it to Aardvark schema
264
+ GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
265
+ ```
266
+
267
+ Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
268
+
269
+ ```ruby
270
+ # You can store this mapping as a JSON or CSV file and load it into a hash
271
+ id_map = {
272
+ 'My Collection 1' => 'institution:my-collection-1',
273
+ 'My Collection 2' => 'institution:my-collection-2'
274
+ }
275
+
276
+ GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
277
+ ```
278
+
256
279
  ## Tests
257
280
 
258
281
  To run the tests, use:
259
282
 
260
283
  ```sh
261
- $ bundle exec rake spec
284
+ bundle exec rake spec
262
285
  ```
263
286
 
264
287
  ## Contributing
@@ -19,21 +19,25 @@ module GeoCombine
19
19
  # Initialize a new harvester
20
20
  # @param ogm_path [String] path to the directory where repositories will be cloned
21
21
  # @param skip_repos [Array<String>] list of repository names to skip
22
+ # @param skip_restricted [Boolean] whether to skip indexing restricted documents
22
23
  # @param schema_version [String] schema version to filter repositories by
23
24
  # @param logger [Logger] logger to use for logging messages
24
25
  def initialize(
25
26
  ogm_path:,
26
27
  skip_repos: ENV.fetch('OGM_SKIP_REPOS', '').split(',').map(&:strip),
28
+ skip_restricted: ENV.fetch('OGM_SKIP_RESTRICTED', 'true') == 'true',
27
29
  schema_version: ENV.fetch('SCHEMA_VERSION', 'Aardvark'),
28
30
  logger: GeoCombine::Logger.logger
29
31
  )
30
32
  @ogm_path = ogm_path
31
33
  @schema_version = schema_version
32
34
  @skip_repos = skip_repos
35
+ @skip_restricted = skip_restricted
33
36
  @logger = logger
34
37
  end
35
38
 
36
39
  # Enumerable of docs to index, for passing to an indexer
40
+ # rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
37
41
  def docs_to_index
38
42
  return to_enum(:docs_to_index) unless block_given?
39
43
 
@@ -49,6 +53,7 @@ module GeoCombine
49
53
  [doc].flatten.each do |record|
50
54
  record_schema = record['gbl_mdVersion_s'] || record['geoblacklight_version']
51
55
  record_id = record['layer_slug_s'] || record['dc_identifier_s']
56
+ record_rights = record['dct_accessRights_s'] || record['dc_rights_s']
52
57
 
53
58
  # skip indexing if no identifiable schema version
54
59
  unless record_schema
@@ -62,11 +67,18 @@ module GeoCombine
62
67
  next
63
68
  end
64
69
 
70
+ # skip indexing if this record is restricted and we want to skip restricted records
71
+ if @skip_restricted && record_rights == 'Restricted'
72
+ @logger.debug "skipping #{record_id}; access rights are restricted"
73
+ next
74
+ end
75
+
65
76
  @logger.debug "found record #{record_id} at #{path}"
66
77
  yield record, path
67
78
  end
68
79
  end
69
80
  end
81
+ # rubocop:enable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
70
82
 
71
83
  # Update a repository via git
72
84
  # If the repository doesn't exist, clone it.
@@ -10,12 +10,19 @@ module GeoCombine
10
10
  class Indexer
11
11
  attr_reader :solr
12
12
 
13
- def initialize(solr: nil, logger: GeoCombine::Logger.logger)
13
+ # Initialize a new indexer
14
+ # @param solr [RSolr::Client] an existing RSolr client
15
+ # @param solr_url [String] the URL of the Solr instance to connect to
16
+ # @param logger [Logger] a logger instance for logging messages
17
+ def initialize(
18
+ solr: nil,
19
+ solr_url: ENV.fetch('SOLR_URL', nil),
20
+ logger: GeoCombine::Logger.logger
21
+ )
14
22
  @logger = logger
15
23
  @batch_size = ENV.fetch('SOLR_BATCH_SIZE', 100).to_i
16
24
 
17
25
  # If SOLR_URL is set, use it; if in a Geoblacklight app, use its solr core
18
- solr_url = ENV.fetch('SOLR_URL', nil)
19
26
  solr_url ||= Blacklight.default_index.connection.base_uri.to_s if defined? Blacklight
20
27
 
21
28
  # If neither, warn and try to use local Blacklight default solr core
@@ -24,6 +31,7 @@ module GeoCombine
24
31
  solr_url = 'http://localhost:8983/solr/blacklight-core'
25
32
  end
26
33
 
34
+ # If solr connection was provided directly, use it; otherwise connect now
27
35
  @solr = solr || RSolr.connect(client, url: solr_url)
28
36
  end
29
37
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module GeoCombine
4
- VERSION = '0.10.0'
4
+ VERSION = '0.11.0'
5
5
  end
@@ -0,0 +1,34 @@
1
+ {
2
+ "dc_publisher_s": "",
3
+ "dc_type_s": "Dataset",
4
+ "dct_title_s": "Property Lot, Arlington County, VA ",
5
+ "dct_description_sm": [
6
+ "This polygon represents property lot in Arlington County, VA"
7
+ ],
8
+ "schema_provider_s": "GMU",
9
+ "gbl_resourceType_sm": [
10
+ "Polygon"
11
+ ],
12
+ "dct_subject_sm": [
13
+ "Property"
14
+ ],
15
+ "gbl_indexYear_im": [
16
+ 1
17
+ ],
18
+ "locn_geometry": "ENVELOPE(-77.1735, -77.0305, 38.9349, 38.8268)",
19
+ "dct_accessRights_s": "Restricted",
20
+ "dct_format_s": "Shapefile",
21
+ "gbl_wxsIdentifier_s": ":arlington_2004_lot_poly",
22
+ "id": "gmu-arlington-2004-lot-poly",
23
+ "dct_identifier_sm": [
24
+ "GMU.arlington_2004_lot_poly"
25
+ ],
26
+ "gbl_mdVersion_s": "Aardvark",
27
+ "dct_temporal_sm": [
28
+ "0001-01-01 01:01:01+00:00"
29
+ ],
30
+ "dct_spatial_sm": [
31
+ "virginia"
32
+ ],
33
+ "dct_references_s": "{\"http://www.opengis.net/def/serviceType/ogc/wfs\":\"http://localhost:8080/secure-geoserver/wfs\",\"http://www.opengis.net/def/serviceType/ogc/wms\":\"http://localhost:8080/secure-geoserver/wms\"}"
34
+ }
@@ -0,0 +1,42 @@
1
+ {
2
+ "dc_creator_sm": [
3
+ "United States. Defense Mapping Agency"
4
+ ],
5
+ "dc_description_s": "This raster layer is an image of a 1:5,000,000 scale aeronautical chart sheet that projects to Lambert Conformal Conic. The National Geospatial-Intelligence Agency and its legacy organizations (Defense Mapping Agency, National Imagery and Mapping Agency) originally published the series, and all sheets are in English. East View Geospatial has completed digitization, georeferencing, and electronic archiving services. To see the map index for this chart series, refer to the Derived Data layer.",
6
+ "dc_format_s": "WorldImage TIFF",
7
+ "dc_identifier_s": "http://hdl.handle.net/2451/38709",
8
+ "dc_language_s": "English",
9
+ "dc_publisher_s": "East View Cartographic, Inc.",
10
+ "dc_rights_s": "Restricted",
11
+ "dc_subject_sm": [
12
+ "Aeronautical charts"
13
+ ],
14
+ "dc_title_s": "Global Navigation and Planning Chart, Sheet 21 W ",
15
+ "dc_type_s": "Dataset",
16
+ "dct_isPartOf_sm": [
17
+ "EVG Open Data Package"
18
+ ],
19
+ "dct_issued_s": "5/31/2016",
20
+ "dct_provenance_s": "NYU",
21
+ "dct_references_s": "{\"http://schema.org/url\":\"http://hdl.handle.net/2451/38709\",\"http://schema.org/downloadUrl\":\"https://archive.nyu.edu/retrieve/84050/nyu_2451_38709.zip\",\"http://www.opengis.net/def/serviceType/ogc/wfs\":\"https://maps-restricted.geo.nyu.edu/geoserver/sdr/wfs\",\"http://www.opengis.net/def/serviceType/ogc/wms\":\"https://maps-restricted.geo.nyu.edu/geoserver/sdr/wms\"}",
22
+ "dct_source_sm": [
23
+ "nyu-2451-38715"
24
+ ],
25
+ "dct_spatial_sm": [
26
+
27
+ ],
28
+ "dct_temporal_sm": [
29
+ "2016"
30
+ ],
31
+ "geoblacklight_version": "1.0",
32
+ "layer_geom_type_s": "Raster",
33
+ "layer_id_s": "sdr:nyu_2451_38709",
34
+ "layer_modified_dt": "2018-01-16T15:11:26Z",
35
+ "layer_slug_s": "nyu-2451-38709",
36
+ "nyu_addl_dspace_s": "39708",
37
+ "nyu_addl_format_sm": [
38
+ "WorldImage TIFF"
39
+ ],
40
+ "solr_geom": "ENVELOPE(-179.99999999999994, -177.98915207217718, -39.99974999999998, -75.0037465617225)",
41
+ "solr_year_i": 2016
42
+ }
@@ -54,6 +54,40 @@ RSpec.describe GeoCombine::Harvester do
54
54
  [JSON.parse(File.read('spec/fixtures/indexing/geoblacklight.json')), 'spec/fixtures/indexing/geoblacklight.json']
55
55
  )
56
56
  end
57
+
58
+ # Collect the titles of every record yielded, regardless of schema field
59
+ def yielded_titles(harvester)
60
+ harvester.docs_to_index.map { |record, _path| record['dct_title_s'] || record['dc_title_s'] }
61
+ end
62
+
63
+ context 'when skip_restricted is true (the default)' do
64
+ it 'skips records with restricted access rights' do
65
+ expect(yielded_titles(harvester)).not_to include('Property Lot, Arlington County, VA ')
66
+ end
67
+
68
+ it 'skips gbl1 records with restricted access rights' do
69
+ harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', schema_version: '1.0', logger:)
70
+ expect(yielded_titles(harvester)).not_to include('Global Navigation and Planning Chart, Sheet 21 W ')
71
+ end
72
+ end
73
+
74
+ context 'when skip_restricted is false' do
75
+ it 'yields records with restricted access rights' do
76
+ harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', skip_restricted: false, logger:)
77
+ expect(yielded_titles(harvester)).to include('Property Lot, Arlington County, VA ')
78
+ end
79
+
80
+ it 'yields gbl1 records with restricted access rights' do
81
+ harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', schema_version: '1.0', skip_restricted: false, logger:)
82
+ expect(yielded_titles(harvester)).to include('Global Navigation and Planning Chart, Sheet 21 W ')
83
+ end
84
+
85
+ it 'can read from the OGM_SKIP_RESTRICTED environment variable' do
86
+ stub_const('ENV', ENV.to_h.merge('OGM_SKIP_RESTRICTED' => 'false'))
87
+ harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', logger:)
88
+ expect(yielded_titles(harvester)).to include('Property Lot, Arlington County, VA ')
89
+ end
90
+ end
57
91
  end
58
92
 
59
93
  describe '#pull' do
metadata CHANGED
@@ -1,13 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: geo_combine
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.0
4
+ version: 0.11.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jack Reed
8
8
  bindir: bin
9
9
  cert_chain: []
10
- date: 2026-07-07 00:00:00.000000000 Z
10
+ date: 2026-07-16 00:00:00.000000000 Z
11
11
  dependencies:
12
12
  - !ruby/object:Gem::Dependency
13
13
  name: activesupport
@@ -340,6 +340,8 @@ files:
340
340
  - spec/fixtures/indexing/basic_geoblacklight.json
341
341
  - spec/fixtures/indexing/geoblacklight.json
342
342
  - spec/fixtures/indexing/layers.json
343
+ - spec/fixtures/indexing/restricted_aardvark.json
344
+ - spec/fixtures/indexing/restricted_geoblacklight.json
343
345
  - spec/fixtures/indexing/test.txt
344
346
  - spec/fixtures/json_docs.rb
345
347
  - spec/fixtures/xml_docs.rb
@@ -402,6 +404,8 @@ test_files:
402
404
  - spec/fixtures/indexing/basic_geoblacklight.json
403
405
  - spec/fixtures/indexing/geoblacklight.json
404
406
  - spec/fixtures/indexing/layers.json
407
+ - spec/fixtures/indexing/restricted_aardvark.json
408
+ - spec/fixtures/indexing/restricted_geoblacklight.json
405
409
  - spec/fixtures/indexing/test.txt
406
410
  - spec/fixtures/json_docs.rb
407
411
  - spec/fixtures/xml_docs.rb