geo_combine 0.10.0 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +114 -91
- data/lib/geo_combine/harvester.rb +12 -0
- data/lib/geo_combine/indexer.rb +10 -2
- data/lib/geo_combine/version.rb +1 -1
- data/spec/fixtures/indexing/restricted_aardvark.json +34 -0
- data/spec/fixtures/indexing/restricted_geoblacklight.json +42 -0
- data/spec/lib/geo_combine/harvester_spec.rb +34 -0
- metadata +6 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 21614ec82bcbe5c5a7bb1291e398795f1baff048552746335f18f315da677c9e
|
|
4
|
+
data.tar.gz: 6c94a18a0f43fd141f5967930a909efac60ab560ac91a0759e48cc45fdab39b5
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 75ce7b76dfa6ad5905410e6b38a18ee98185c50d5d116c8fc5794eba3376e85a05d2e2ad7d70d6bde8de7b0073df22f15cb28759633e0d3c6e546e24a13dee82
|
|
7
|
+
data.tar.gz: cbd5ccc9eac444e599e0dce59fda1f23ef752cec71508a83fbe32ea1b2785b5a8c52d66c4ebcebc81a913aaccf6f677f5555622a43a8e16e902ab5f3379cbfa0
|
data/README.md
CHANGED
|
@@ -1,16 +1,20 @@
|
|
|
1
1
|
# GeoCombine
|
|
2
2
|
|
|
3
3
|

|
|
4
|
-
|
|
5
|
-
|
|
4
|
+
[]()
|
|
5
|
+
[](https://github.com/OpenGeoMetadata/GeoCombine/releases)
|
|
6
6
|
|
|
7
7
|
A Ruby toolkit for managing geospatial metadata, including:
|
|
8
8
|
|
|
9
|
-
-
|
|
10
|
-
-
|
|
9
|
+
- harvesting and indexing records from OpenGeoMetadata
|
|
10
|
+
- harvesting records directly from a GeoBlacklight instance
|
|
11
|
+
- converting geospatial metadata records between standards
|
|
11
12
|
|
|
12
13
|
## Installation
|
|
13
14
|
|
|
15
|
+
> [!TIP]
|
|
16
|
+
> If you're using GeoBlacklight, GeoCombine is already bundled with your installation – you can skip this section.
|
|
17
|
+
|
|
14
18
|
Add this line to your application's `Gemfile`:
|
|
15
19
|
|
|
16
20
|
```ruby
|
|
@@ -20,127 +24,64 @@ gem 'geo_combine'
|
|
|
20
24
|
And then execute:
|
|
21
25
|
|
|
22
26
|
```sh
|
|
23
|
-
|
|
27
|
+
bundle install
|
|
24
28
|
```
|
|
25
29
|
|
|
26
30
|
Or install it yourself as:
|
|
27
31
|
|
|
28
32
|
```sh
|
|
29
|
-
|
|
33
|
+
gem install geo_combine
|
|
30
34
|
```
|
|
31
35
|
|
|
32
36
|
## Usage
|
|
33
37
|
|
|
34
|
-
### Converting metadata
|
|
35
|
-
|
|
36
|
-
#### Converting metadata into GeoBlacklight JSON
|
|
37
|
-
|
|
38
|
-
GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
|
|
39
|
-
|
|
40
|
-
```ruby
|
|
41
|
-
GeoCombine::Iso19139 # ISO 19139 XML
|
|
42
|
-
GeoCombine::OGP # OpenGeoPortal JSON
|
|
43
|
-
GeoCombine::Fgdc # FGDC XML
|
|
44
|
-
GeoCombine::EsriOpenData # Esri Open Data Portal JSON
|
|
45
|
-
GeoCombine::CkanMetadata # CKAN JSON
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
An example for converting an ISO 19139 XML record:
|
|
49
|
-
|
|
50
|
-
```ruby
|
|
51
|
-
# Create a new ISO19139 object
|
|
52
|
-
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
|
|
53
|
-
|
|
54
|
-
# Convert to GeoBlacklight's metadata format
|
|
55
|
-
> iso_metadata.to_geoblacklight
|
|
56
|
-
|
|
57
|
-
# Output it as JSON instead of a Ruby hash
|
|
58
|
-
> iso_metadata.to_geoblacklight.to_json
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
Some formats also support conversion into HTML for display in a web browser:
|
|
62
|
-
|
|
63
|
-
```ruby
|
|
64
|
-
# Create a new ISO19139 object
|
|
65
|
-
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
|
|
66
|
-
|
|
67
|
-
# Convert ISO to HTML
|
|
68
|
-
> iso_metadata.to_html
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
### Migrating metadata
|
|
72
|
-
|
|
73
|
-
You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
|
|
74
|
-
|
|
75
|
-
Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
|
|
76
|
-
|
|
77
|
-
```ruby
|
|
78
|
-
# Load a record in geoblacklight v1 schema
|
|
79
|
-
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
|
|
80
|
-
|
|
81
|
-
# Migrate it to Aardvark schema
|
|
82
|
-
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
|
|
86
|
-
|
|
87
|
-
```ruby
|
|
88
|
-
# You can store this mapping as a JSON or CSV file and load it into a hash
|
|
89
|
-
id_map = {
|
|
90
|
-
'My Collection 1' => 'institution:my-collection-1',
|
|
91
|
-
'My Collection 2' => 'institution:my-collection-2'
|
|
92
|
-
}
|
|
93
|
-
|
|
94
|
-
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
|
|
95
|
-
```
|
|
96
|
-
|
|
97
38
|
### Downloading metadata from OpenGeoMetadata
|
|
98
39
|
|
|
99
|
-
####
|
|
40
|
+
#### Schema version
|
|
100
41
|
|
|
101
|
-
|
|
42
|
+
By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the `SCHEMA_VERSION` environment variable:
|
|
102
43
|
|
|
103
44
|
```sh
|
|
104
|
-
|
|
45
|
+
# Only fetch and index schema version 1.0 records
|
|
46
|
+
export SCHEMA_VERSION=1.0
|
|
105
47
|
```
|
|
106
48
|
|
|
107
|
-
####
|
|
49
|
+
#### Restricted records
|
|
108
50
|
|
|
109
|
-
By default, GeoCombine will only fetch and index records
|
|
51
|
+
By default, GeoCombine will only fetch and index records that are not marked as restricted. If you want to include restricted records, you can set the `OGM_SKIP_RESTRICTED` environment variable to `false`:
|
|
110
52
|
|
|
111
53
|
```sh
|
|
112
|
-
|
|
113
|
-
$ export SCHEMA_VERSION=1.0
|
|
54
|
+
export OGM_SKIP_RESTRICTED=false
|
|
114
55
|
```
|
|
115
56
|
|
|
116
57
|
#### Clone OpenGeoMetadata repositories locally
|
|
117
58
|
|
|
118
59
|
```sh
|
|
119
|
-
|
|
60
|
+
bundle exec rake geocombine:clone
|
|
120
61
|
```
|
|
121
62
|
|
|
122
63
|
Will clone all OpenGeoMetadata repositories containing metadata matching the `SCHEMA_VERSION` into `./tmp/opengeometadata`. Location of the OpenGeoMetadata repositories can be configured using the `OGM_PATH` environment variable.
|
|
123
64
|
|
|
124
65
|
```sh
|
|
125
|
-
|
|
66
|
+
OGM_PATH='my/custom/location' bundle exec rake geocombine:clone
|
|
126
67
|
```
|
|
127
68
|
|
|
128
69
|
You can also specify a single repository:
|
|
129
70
|
|
|
130
71
|
```sh
|
|
131
|
-
|
|
72
|
+
bundle exec rake geocombine:clone[edu.stanford.purl]
|
|
132
73
|
```
|
|
133
74
|
|
|
134
75
|
_Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
|
|
135
76
|
|
|
136
77
|
```sh
|
|
137
|
-
|
|
78
|
+
bundle exec rake geocombine:clone\[edu.stanford.purl\]
|
|
138
79
|
```
|
|
139
80
|
|
|
140
81
|
#### Update local OpenGeoMetadata repositories
|
|
141
82
|
|
|
142
83
|
```sh
|
|
143
|
-
|
|
84
|
+
bundle exec rake geocombine:pull
|
|
144
85
|
```
|
|
145
86
|
|
|
146
87
|
Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometadata` (or custom path with configured environment variable `OGM_PATH`).
|
|
@@ -148,22 +89,25 @@ Runs `git pull origin master` on all cloned repositories in `./tmp/opengeometada
|
|
|
148
89
|
You can also specify a single repository:
|
|
149
90
|
|
|
150
91
|
```sh
|
|
151
|
-
|
|
92
|
+
bundle exec rake geocombine:pull[edu.stanford.purl]
|
|
152
93
|
```
|
|
153
94
|
|
|
154
95
|
_Note: If you are using zsh, you will need to use escape characters in front of the brackets:_
|
|
155
96
|
|
|
156
97
|
```sh
|
|
157
|
-
|
|
98
|
+
bundle exec rake geocombine:pull\[edu.stanford.purl\]
|
|
158
99
|
```
|
|
159
100
|
|
|
160
101
|
#### Index GeoBlacklight documents
|
|
161
102
|
|
|
103
|
+
> [!TIP]
|
|
104
|
+
> If you're using GeoBlacklight, GeoCombine will automatically detect the path to your Solr index if you run it in the directory where GeoBlacklight is installed.
|
|
105
|
+
|
|
162
106
|
To index into Solr, GeoCombine requires a Solr instance that is running the
|
|
163
107
|
[GeoBlacklight schema](https://github.com/geoblacklight/geoblacklight):
|
|
164
108
|
|
|
165
109
|
```sh
|
|
166
|
-
|
|
110
|
+
bundle exec rake geocombine:index
|
|
167
111
|
```
|
|
168
112
|
|
|
169
113
|
If Blacklight is installed in the ruby environment and a solr index is configured, the rake task will use the solr index configured in the Blacklight application (this is the case when invoking GeoCombine from your GeoBlacklight installation). If Blacklight is unavailable, the rake task will try to find a Solr instance running at `http://localhost:8983/solr/blacklight-core`.
|
|
@@ -171,24 +115,40 @@ If Blacklight is installed in the ruby environment and a solr index is configure
|
|
|
171
115
|
You can also set the Solr instance URL using `SOLR_URL`:
|
|
172
116
|
|
|
173
117
|
```sh
|
|
174
|
-
|
|
118
|
+
SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
#### Logging
|
|
122
|
+
|
|
123
|
+
Some of the tools and scripts in this gem use Ruby's `Logger` class to print information to `$stderr`. By default, the log level is set to `Logger::INFO`. For more verbose information, you can set the `LOG_LEVEL` environment variable to `DEBUG`:
|
|
124
|
+
|
|
125
|
+
```sh
|
|
126
|
+
export LOG_LEVEL=DEBUG
|
|
175
127
|
```
|
|
176
128
|
|
|
177
129
|
### Indexing local documents
|
|
178
130
|
|
|
179
|
-
To index
|
|
131
|
+
To index a collection of records in a local directory, you can pass the directory as a `rake` argument:
|
|
132
|
+
|
|
133
|
+
```sh
|
|
134
|
+
bundle exec rake geocombine:index\[/path/to/your/files\]
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Or you can provide it as `OGM_PATH`:
|
|
180
138
|
|
|
181
139
|
```sh
|
|
182
|
-
rake geocombine:index\[/path/to/your/files\]
|
|
183
140
|
OGM_PATH=/path/to/your/files rake geocombine:index
|
|
184
141
|
```
|
|
185
142
|
|
|
186
|
-
|
|
143
|
+
For a single file, it works the same way:
|
|
187
144
|
|
|
188
|
-
|
|
145
|
+
```sh
|
|
146
|
+
bundle exec rake geocombine:index\[/path/to/your/file.json\]
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
Or:
|
|
189
150
|
|
|
190
151
|
```sh
|
|
191
|
-
rake geocombine:index\[/path/to/your/file.json\]
|
|
192
152
|
OGM_PATH=/path/to/your/file.json rake geocombine:index
|
|
193
153
|
```
|
|
194
154
|
|
|
@@ -253,12 +213,75 @@ GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
|
|
|
253
213
|
end
|
|
254
214
|
```
|
|
255
215
|
|
|
216
|
+
### Converting metadata
|
|
217
|
+
|
|
218
|
+
#### Converting metadata into GeoBlacklight JSON
|
|
219
|
+
|
|
220
|
+
GeoCombine provides several classes representing different metadata standards that implement the `#to_geoblacklight` method for generating records in the [GeoBlacklight JSON format](https://opengeometadata.org/reference/):
|
|
221
|
+
|
|
222
|
+
```ruby
|
|
223
|
+
GeoCombine::Iso19139 # ISO 19139 XML
|
|
224
|
+
GeoCombine::OGP # OpenGeoPortal JSON
|
|
225
|
+
GeoCombine::Fgdc # FGDC XML
|
|
226
|
+
GeoCombine::EsriOpenData # Esri Open Data Portal JSON
|
|
227
|
+
GeoCombine::CkanMetadata # CKAN JSON
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
An example for converting an ISO 19139 XML record:
|
|
231
|
+
|
|
232
|
+
```ruby
|
|
233
|
+
# Create a new ISO19139 object
|
|
234
|
+
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
|
|
235
|
+
|
|
236
|
+
# Convert to GeoBlacklight's metadata format
|
|
237
|
+
> iso_metadata.to_geoblacklight
|
|
238
|
+
|
|
239
|
+
# Output it as JSON instead of a Ruby hash
|
|
240
|
+
> iso_metadata.to_geoblacklight.to_json
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
Some formats also support conversion into HTML for display in a web browser:
|
|
244
|
+
|
|
245
|
+
```ruby
|
|
246
|
+
# Create a new ISO19139 object
|
|
247
|
+
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
|
|
248
|
+
|
|
249
|
+
# Convert ISO to HTML
|
|
250
|
+
> iso_metadata.to_html
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
### Migrating metadata
|
|
254
|
+
|
|
255
|
+
You can use the `GeoCombine::Migrators` to migrate metadata from one schema to another.
|
|
256
|
+
|
|
257
|
+
Currently, the only migrator is `GeoCombine::Migrators::V1AardvarkMigrator` which migrates from the [GeoBlacklight v1 schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/gbl-1.0.md) to the [Aardvark schema](https://github.com/OpenGeoMetadata/opengeometadata.github.io/blob/main/docs/ogm-aardvark.md)
|
|
258
|
+
|
|
259
|
+
```ruby
|
|
260
|
+
# Load a record in geoblacklight v1 schema
|
|
261
|
+
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
|
|
262
|
+
|
|
263
|
+
# Migrate it to Aardvark schema
|
|
264
|
+
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
|
|
268
|
+
|
|
269
|
+
```ruby
|
|
270
|
+
# You can store this mapping as a JSON or CSV file and load it into a hash
|
|
271
|
+
id_map = {
|
|
272
|
+
'My Collection 1' => 'institution:my-collection-1',
|
|
273
|
+
'My Collection 2' => 'institution:my-collection-2'
|
|
274
|
+
}
|
|
275
|
+
|
|
276
|
+
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run
|
|
277
|
+
```
|
|
278
|
+
|
|
256
279
|
## Tests
|
|
257
280
|
|
|
258
281
|
To run the tests, use:
|
|
259
282
|
|
|
260
283
|
```sh
|
|
261
|
-
|
|
284
|
+
bundle exec rake spec
|
|
262
285
|
```
|
|
263
286
|
|
|
264
287
|
## Contributing
|
|
@@ -19,21 +19,25 @@ module GeoCombine
|
|
|
19
19
|
# Initialize a new harvester
|
|
20
20
|
# @param ogm_path [String] path to the directory where repositories will be cloned
|
|
21
21
|
# @param skip_repos [Array<String>] list of repository names to skip
|
|
22
|
+
# @param skip_restricted [Boolean] whether to skip indexing restricted documents
|
|
22
23
|
# @param schema_version [String] schema version to filter repositories by
|
|
23
24
|
# @param logger [Logger] logger to use for logging messages
|
|
24
25
|
def initialize(
|
|
25
26
|
ogm_path:,
|
|
26
27
|
skip_repos: ENV.fetch('OGM_SKIP_REPOS', '').split(',').map(&:strip),
|
|
28
|
+
skip_restricted: ENV.fetch('OGM_SKIP_RESTRICTED', 'true') == 'true',
|
|
27
29
|
schema_version: ENV.fetch('SCHEMA_VERSION', 'Aardvark'),
|
|
28
30
|
logger: GeoCombine::Logger.logger
|
|
29
31
|
)
|
|
30
32
|
@ogm_path = ogm_path
|
|
31
33
|
@schema_version = schema_version
|
|
32
34
|
@skip_repos = skip_repos
|
|
35
|
+
@skip_restricted = skip_restricted
|
|
33
36
|
@logger = logger
|
|
34
37
|
end
|
|
35
38
|
|
|
36
39
|
# Enumerable of docs to index, for passing to an indexer
|
|
40
|
+
# rubocop:disable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
37
41
|
def docs_to_index
|
|
38
42
|
return to_enum(:docs_to_index) unless block_given?
|
|
39
43
|
|
|
@@ -49,6 +53,7 @@ module GeoCombine
|
|
|
49
53
|
[doc].flatten.each do |record|
|
|
50
54
|
record_schema = record['gbl_mdVersion_s'] || record['geoblacklight_version']
|
|
51
55
|
record_id = record['layer_slug_s'] || record['dc_identifier_s']
|
|
56
|
+
record_rights = record['dct_accessRights_s'] || record['dc_rights_s']
|
|
52
57
|
|
|
53
58
|
# skip indexing if no identifiable schema version
|
|
54
59
|
unless record_schema
|
|
@@ -62,11 +67,18 @@ module GeoCombine
|
|
|
62
67
|
next
|
|
63
68
|
end
|
|
64
69
|
|
|
70
|
+
# skip indexing if this record is restricted and we want to skip restricted records
|
|
71
|
+
if @skip_restricted && record_rights == 'Restricted'
|
|
72
|
+
@logger.debug "skipping #{record_id}; access rights are restricted"
|
|
73
|
+
next
|
|
74
|
+
end
|
|
75
|
+
|
|
65
76
|
@logger.debug "found record #{record_id} at #{path}"
|
|
66
77
|
yield record, path
|
|
67
78
|
end
|
|
68
79
|
end
|
|
69
80
|
end
|
|
81
|
+
# rubocop:enable Metrics/MethodLength, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
70
82
|
|
|
71
83
|
# Update a repository via git
|
|
72
84
|
# If the repository doesn't exist, clone it.
|
data/lib/geo_combine/indexer.rb
CHANGED
|
@@ -10,12 +10,19 @@ module GeoCombine
|
|
|
10
10
|
class Indexer
|
|
11
11
|
attr_reader :solr
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
# Initialize a new indexer
|
|
14
|
+
# @param solr [RSolr::Client] an existing RSolr client
|
|
15
|
+
# @param solr_url [String] the URL of the Solr instance to connect to
|
|
16
|
+
# @param logger [Logger] a logger instance for logging messages
|
|
17
|
+
def initialize(
|
|
18
|
+
solr: nil,
|
|
19
|
+
solr_url: ENV.fetch('SOLR_URL', nil),
|
|
20
|
+
logger: GeoCombine::Logger.logger
|
|
21
|
+
)
|
|
14
22
|
@logger = logger
|
|
15
23
|
@batch_size = ENV.fetch('SOLR_BATCH_SIZE', 100).to_i
|
|
16
24
|
|
|
17
25
|
# If SOLR_URL is set, use it; if in a Geoblacklight app, use its solr core
|
|
18
|
-
solr_url = ENV.fetch('SOLR_URL', nil)
|
|
19
26
|
solr_url ||= Blacklight.default_index.connection.base_uri.to_s if defined? Blacklight
|
|
20
27
|
|
|
21
28
|
# If neither, warn and try to use local Blacklight default solr core
|
|
@@ -24,6 +31,7 @@ module GeoCombine
|
|
|
24
31
|
solr_url = 'http://localhost:8983/solr/blacklight-core'
|
|
25
32
|
end
|
|
26
33
|
|
|
34
|
+
# If solr connection was provided directly, use it; otherwise connect now
|
|
27
35
|
@solr = solr || RSolr.connect(client, url: solr_url)
|
|
28
36
|
end
|
|
29
37
|
|
data/lib/geo_combine/version.rb
CHANGED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
{
|
|
2
|
+
"dc_publisher_s": "",
|
|
3
|
+
"dc_type_s": "Dataset",
|
|
4
|
+
"dct_title_s": "Property Lot, Arlington County, VA ",
|
|
5
|
+
"dct_description_sm": [
|
|
6
|
+
"This polygon represents property lot in Arlington County, VA"
|
|
7
|
+
],
|
|
8
|
+
"schema_provider_s": "GMU",
|
|
9
|
+
"gbl_resourceType_sm": [
|
|
10
|
+
"Polygon"
|
|
11
|
+
],
|
|
12
|
+
"dct_subject_sm": [
|
|
13
|
+
"Property"
|
|
14
|
+
],
|
|
15
|
+
"gbl_indexYear_im": [
|
|
16
|
+
1
|
|
17
|
+
],
|
|
18
|
+
"locn_geometry": "ENVELOPE(-77.1735, -77.0305, 38.9349, 38.8268)",
|
|
19
|
+
"dct_accessRights_s": "Restricted",
|
|
20
|
+
"dct_format_s": "Shapefile",
|
|
21
|
+
"gbl_wxsIdentifier_s": ":arlington_2004_lot_poly",
|
|
22
|
+
"id": "gmu-arlington-2004-lot-poly",
|
|
23
|
+
"dct_identifier_sm": [
|
|
24
|
+
"GMU.arlington_2004_lot_poly"
|
|
25
|
+
],
|
|
26
|
+
"gbl_mdVersion_s": "Aardvark",
|
|
27
|
+
"dct_temporal_sm": [
|
|
28
|
+
"0001-01-01 01:01:01+00:00"
|
|
29
|
+
],
|
|
30
|
+
"dct_spatial_sm": [
|
|
31
|
+
"virginia"
|
|
32
|
+
],
|
|
33
|
+
"dct_references_s": "{\"http://www.opengis.net/def/serviceType/ogc/wfs\":\"http://localhost:8080/secure-geoserver/wfs\",\"http://www.opengis.net/def/serviceType/ogc/wms\":\"http://localhost:8080/secure-geoserver/wms\"}"
|
|
34
|
+
}
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
{
|
|
2
|
+
"dc_creator_sm": [
|
|
3
|
+
"United States. Defense Mapping Agency"
|
|
4
|
+
],
|
|
5
|
+
"dc_description_s": "This raster layer is an image of a 1:5,000,000 scale aeronautical chart sheet that projects to Lambert Conformal Conic. The National Geospatial-Intelligence Agency and its legacy organizations (Defense Mapping Agency, National Imagery and Mapping Agency) originally published the series, and all sheets are in English. East View Geospatial has completed digitization, georeferencing, and electronic archiving services. To see the map index for this chart series, refer to the Derived Data layer.",
|
|
6
|
+
"dc_format_s": "WorldImage TIFF",
|
|
7
|
+
"dc_identifier_s": "http://hdl.handle.net/2451/38709",
|
|
8
|
+
"dc_language_s": "English",
|
|
9
|
+
"dc_publisher_s": "East View Cartographic, Inc.",
|
|
10
|
+
"dc_rights_s": "Restricted",
|
|
11
|
+
"dc_subject_sm": [
|
|
12
|
+
"Aeronautical charts"
|
|
13
|
+
],
|
|
14
|
+
"dc_title_s": "Global Navigation and Planning Chart, Sheet 21 W ",
|
|
15
|
+
"dc_type_s": "Dataset",
|
|
16
|
+
"dct_isPartOf_sm": [
|
|
17
|
+
"EVG Open Data Package"
|
|
18
|
+
],
|
|
19
|
+
"dct_issued_s": "5/31/2016",
|
|
20
|
+
"dct_provenance_s": "NYU",
|
|
21
|
+
"dct_references_s": "{\"http://schema.org/url\":\"http://hdl.handle.net/2451/38709\",\"http://schema.org/downloadUrl\":\"https://archive.nyu.edu/retrieve/84050/nyu_2451_38709.zip\",\"http://www.opengis.net/def/serviceType/ogc/wfs\":\"https://maps-restricted.geo.nyu.edu/geoserver/sdr/wfs\",\"http://www.opengis.net/def/serviceType/ogc/wms\":\"https://maps-restricted.geo.nyu.edu/geoserver/sdr/wms\"}",
|
|
22
|
+
"dct_source_sm": [
|
|
23
|
+
"nyu-2451-38715"
|
|
24
|
+
],
|
|
25
|
+
"dct_spatial_sm": [
|
|
26
|
+
|
|
27
|
+
],
|
|
28
|
+
"dct_temporal_sm": [
|
|
29
|
+
"2016"
|
|
30
|
+
],
|
|
31
|
+
"geoblacklight_version": "1.0",
|
|
32
|
+
"layer_geom_type_s": "Raster",
|
|
33
|
+
"layer_id_s": "sdr:nyu_2451_38709",
|
|
34
|
+
"layer_modified_dt": "2018-01-16T15:11:26Z",
|
|
35
|
+
"layer_slug_s": "nyu-2451-38709",
|
|
36
|
+
"nyu_addl_dspace_s": "39708",
|
|
37
|
+
"nyu_addl_format_sm": [
|
|
38
|
+
"WorldImage TIFF"
|
|
39
|
+
],
|
|
40
|
+
"solr_geom": "ENVELOPE(-179.99999999999994, -177.98915207217718, -39.99974999999998, -75.0037465617225)",
|
|
41
|
+
"solr_year_i": 2016
|
|
42
|
+
}
|
|
@@ -54,6 +54,40 @@ RSpec.describe GeoCombine::Harvester do
|
|
|
54
54
|
[JSON.parse(File.read('spec/fixtures/indexing/geoblacklight.json')), 'spec/fixtures/indexing/geoblacklight.json']
|
|
55
55
|
)
|
|
56
56
|
end
|
|
57
|
+
|
|
58
|
+
# Collect the titles of every record yielded, regardless of schema field
|
|
59
|
+
def yielded_titles(harvester)
|
|
60
|
+
harvester.docs_to_index.map { |record, _path| record['dct_title_s'] || record['dc_title_s'] }
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
context 'when skip_restricted is true (the default)' do
|
|
64
|
+
it 'skips records with restricted access rights' do
|
|
65
|
+
expect(yielded_titles(harvester)).not_to include('Property Lot, Arlington County, VA ')
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
it 'skips gbl1 records with restricted access rights' do
|
|
69
|
+
harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', schema_version: '1.0', logger:)
|
|
70
|
+
expect(yielded_titles(harvester)).not_to include('Global Navigation and Planning Chart, Sheet 21 W ')
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
context 'when skip_restricted is false' do
|
|
75
|
+
it 'yields records with restricted access rights' do
|
|
76
|
+
harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', skip_restricted: false, logger:)
|
|
77
|
+
expect(yielded_titles(harvester)).to include('Property Lot, Arlington County, VA ')
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
it 'yields gbl1 records with restricted access rights' do
|
|
81
|
+
harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', schema_version: '1.0', skip_restricted: false, logger:)
|
|
82
|
+
expect(yielded_titles(harvester)).to include('Global Navigation and Planning Chart, Sheet 21 W ')
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
it 'can read from the OGM_SKIP_RESTRICTED environment variable' do
|
|
86
|
+
stub_const('ENV', ENV.to_h.merge('OGM_SKIP_RESTRICTED' => 'false'))
|
|
87
|
+
harvester = described_class.new(ogm_path: 'spec/fixtures/indexing', logger:)
|
|
88
|
+
expect(yielded_titles(harvester)).to include('Property Lot, Arlington County, VA ')
|
|
89
|
+
end
|
|
90
|
+
end
|
|
57
91
|
end
|
|
58
92
|
|
|
59
93
|
describe '#pull' do
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: geo_combine
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.11.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Jack Reed
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date: 2026-07-
|
|
10
|
+
date: 2026-07-16 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: activesupport
|
|
@@ -340,6 +340,8 @@ files:
|
|
|
340
340
|
- spec/fixtures/indexing/basic_geoblacklight.json
|
|
341
341
|
- spec/fixtures/indexing/geoblacklight.json
|
|
342
342
|
- spec/fixtures/indexing/layers.json
|
|
343
|
+
- spec/fixtures/indexing/restricted_aardvark.json
|
|
344
|
+
- spec/fixtures/indexing/restricted_geoblacklight.json
|
|
343
345
|
- spec/fixtures/indexing/test.txt
|
|
344
346
|
- spec/fixtures/json_docs.rb
|
|
345
347
|
- spec/fixtures/xml_docs.rb
|
|
@@ -402,6 +404,8 @@ test_files:
|
|
|
402
404
|
- spec/fixtures/indexing/basic_geoblacklight.json
|
|
403
405
|
- spec/fixtures/indexing/geoblacklight.json
|
|
404
406
|
- spec/fixtures/indexing/layers.json
|
|
407
|
+
- spec/fixtures/indexing/restricted_aardvark.json
|
|
408
|
+
- spec/fixtures/indexing/restricted_geoblacklight.json
|
|
405
409
|
- spec/fixtures/indexing/test.txt
|
|
406
410
|
- spec/fixtures/json_docs.rb
|
|
407
411
|
- spec/fixtures/xml_docs.rb
|