overturemaps 0.2.0__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,68 @@
1
+ Metadata-Version: 2.1
2
+ Name: overturemaps
3
+ Version: 0.3.0
4
+ Summary: Python tools for interacting with Overture Maps (overturemaps.org) data.
5
+ License: MIT
6
+ Author: Jacob Wasserman
7
+ Author-email: jwasserman@meta.com
8
+ Requires-Python: >=3.11,<4.0
9
+ Classifier: License :: OSI Approved :: MIT License
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: Programming Language :: Python :: 3.11
12
+ Classifier: Programming Language :: Python :: 3.12
13
+ Requires-Dist: click (>=8.1.7,<9.0.0)
14
+ Requires-Dist: pyarrow (>=15.0.2,<16.0.0)
15
+ Requires-Dist: shapely (>=2.0.3,<3.0.0)
16
+ Description-Content-Type: text/markdown
17
+
18
+ # overturemaps-py
19
+
20
+ Official Python command-line tool of the [Overture Maps Foundation](overturemaps.org)
21
+
22
+ Overture Maps provides free and open geospatial map data, from many different sources and normalized to a
23
+ [common schema](https://github.com/OvertureMaps/schema). This tool helps to download Overture data
24
+ within a region of interest and converts it to a few different file formats. For more information about accessing
25
+ Overture Maps data, see our official documentation site https://docs.overturemaps.org.
26
+
27
+ Note: This repository and project are experimental. Things are likely change including the user interface
28
+ until a stable release, but we will keep the documentation here up-to-date.
29
+
30
+ ## Quick Start
31
+
32
+ Download the building footprints for the specific bounding box as GeoJSON and save to a file named "boston.geojson"
33
+
34
+ ```
35
+ $ overturemaps download --bbox=-71.068,42.353,-71.058,42.363 -f geojson --type=building -o boston.geojson
36
+ ```
37
+
38
+ ## Usage
39
+
40
+ #### `download`
41
+ There is currently one option to the `overturemaps` utility, `download`. It will download Overture Maps data
42
+ with an optional bounding box into the specified file format. When specifying a bounding box,
43
+ only the minimum data is transferred. The result is streamed out and can handle arbitrarily
44
+ large bounding boxes.
45
+
46
+ Command-line options:
47
+ * `--bbox` (optional): west, south, east, north longitude and latitude coordinates. When omitted the
48
+ entire dataset for the specified type will be downloaded
49
+ * `-f` (required: one of "geojson", "geojsonseq", "geoparquet"): output format
50
+ * `--output`/`-o` (optional): Location of output file. When omitted output will be written to stdout.
51
+ * `--type`/`-t` (required): The Overture map data type to be downloaded. Examples of types are `building`
52
+ for building footprints, `place` for POI places data, etc. Run `overturemaps download --help` for the
53
+ complete list of allowed types
54
+
55
+ This downloads data directly from Overture's S3 bucket without interacting with any other servers.
56
+ By including bounding box extents on each row in the Overture distribution, the underlying Parquet
57
+ readers use the Parquet summary statistics to download the minimum amount of data
58
+ necessary to extract data from the desired region.
59
+
60
+ To help find bounding boxes of interest, we like this [bounding box tool](https://boundingbox.klokantech.com/)
61
+ from [Klokantech](https://www.klokantech.com/). Choose the CSV format and copy the value directly into
62
+ the `--bbox` field here.
63
+
64
+
65
+ ## Installation
66
+
67
+ `pip install overturemaps`
68
+
@@ -0,0 +1,50 @@
1
+ # overturemaps-py
2
+
3
+ Official Python command-line tool of the [Overture Maps Foundation](overturemaps.org)
4
+
5
+ Overture Maps provides free and open geospatial map data, from many different sources and normalized to a
6
+ [common schema](https://github.com/OvertureMaps/schema). This tool helps to download Overture data
7
+ within a region of interest and converts it to a few different file formats. For more information about accessing
8
+ Overture Maps data, see our official documentation site https://docs.overturemaps.org.
9
+
10
+ Note: This repository and project are experimental. Things are likely change including the user interface
11
+ until a stable release, but we will keep the documentation here up-to-date.
12
+
13
+ ## Quick Start
14
+
15
+ Download the building footprints for the specific bounding box as GeoJSON and save to a file named "boston.geojson"
16
+
17
+ ```
18
+ $ overturemaps download --bbox=-71.068,42.353,-71.058,42.363 -f geojson --type=building -o boston.geojson
19
+ ```
20
+
21
+ ## Usage
22
+
23
+ #### `download`
24
+ There is currently one option to the `overturemaps` utility, `download`. It will download Overture Maps data
25
+ with an optional bounding box into the specified file format. When specifying a bounding box,
26
+ only the minimum data is transferred. The result is streamed out and can handle arbitrarily
27
+ large bounding boxes.
28
+
29
+ Command-line options:
30
+ * `--bbox` (optional): west, south, east, north longitude and latitude coordinates. When omitted the
31
+ entire dataset for the specified type will be downloaded
32
+ * `-f` (required: one of "geojson", "geojsonseq", "geoparquet"): output format
33
+ * `--output`/`-o` (optional): Location of output file. When omitted output will be written to stdout.
34
+ * `--type`/`-t` (required): The Overture map data type to be downloaded. Examples of types are `building`
35
+ for building footprints, `place` for POI places data, etc. Run `overturemaps download --help` for the
36
+ complete list of allowed types
37
+
38
+ This downloads data directly from Overture's S3 bucket without interacting with any other servers.
39
+ By including bounding box extents on each row in the Overture distribution, the underlying Parquet
40
+ readers use the Parquet summary statistics to download the minimum amount of data
41
+ necessary to extract data from the desired region.
42
+
43
+ To help find bounding boxes of interest, we like this [bounding box tool](https://boundingbox.klokantech.com/)
44
+ from [Klokantech](https://www.klokantech.com/). Choose the CSV format and copy the value directly into
45
+ the `--bbox` field here.
46
+
47
+
48
+ ## Installation
49
+
50
+ `pip install overturemaps`
@@ -0,0 +1,4 @@
1
+ from .core import (
2
+ record_batch_reader,
3
+ get_all_overture_types,
4
+ )
@@ -18,57 +18,7 @@ import pyarrow.fs as fs
18
18
  import pyarrow.parquet as pq
19
19
  import shapely.wkb
20
20
 
21
-
22
- # Map of sub-partition "type" to parent partition "theme" for forming the
23
- # complete s3 path. Could be discovered by reading from the top-level s3
24
- # location but this allows to only read the files in the necessary partition.
25
- _theme_type_mapping = {
26
- "locality": "admins",
27
- "locality_area": "admins",
28
- "administrative_boundary": "admins",
29
- "building": "buildings",
30
- "part": "buildings",
31
- "place": "places",
32
- "segment": "transportation",
33
- "connector": "transportation",
34
- "land": "base",
35
- "land_use": "base",
36
- "water": "base",
37
- }
38
-
39
-
40
- def _dataset_path(overture_type: str) -> str:
41
- """
42
- Returns the s3 path of the Overture dataset to use. This assumes overture_type has
43
- been validated, e.g. by the CLI
44
-
45
- """
46
- theme = _theme_type_mapping[overture_type]
47
- return f"overturemaps-us-west-2/release/2024-03-12-alpha.0/theme={theme}/type={overture_type}/"
48
-
49
-
50
- def record_batch_reader(path, bbox=None) -> Optional[pa.RecordBatchReader]:
51
- """
52
- Return a pyarrow RecordBatchReader for the desired bounding box and s3 path
53
- """
54
- if bbox:
55
- xmin, ymin, xmax, ymax = bbox
56
- filter = (
57
- (pc.field("bbox", "minx") < xmax)
58
- & (pc.field("bbox", "maxx") > xmin)
59
- & (pc.field("bbox", "miny") < ymax)
60
- & (pc.field("bbox", "maxy") > ymin)
61
- )
62
- else:
63
- filter = None
64
-
65
- dataset = ds.dataset(
66
- path, filesystem=fs.S3FileSystem(anonymous=True, region="us-west-2")
67
- )
68
- batches = dataset.to_batches(filter=filter)
69
-
70
- reader = pa.RecordBatchReader.from_batches(dataset.schema, batches)
71
- return reader
21
+ from . core import record_batch_reader, get_all_overture_types
72
22
 
73
23
 
74
24
  def get_writer(output_format, path, schema):
@@ -103,7 +53,9 @@ class BboxParamType(click.ParamType):
103
53
  fail = True
104
54
 
105
55
  if fail or len(bbox) != 4:
106
- self.fail(f"bbox must be 4 floating point numbers separated by commas. Got '{value}'")
56
+ self.fail(
57
+ f"bbox must be 4 floating point numbers separated by commas. Got '{value}'"
58
+ )
107
59
 
108
60
  return bbox
109
61
 
@@ -115,21 +67,25 @@ def cli():
115
67
 
116
68
  @cli.command()
117
69
  @click.option("--bbox", required=False, type=BboxParamType())
118
- @click.option("-f", "output_format", type=click.Choice(["geojson", "geojsonseq", "geoparquet"]), required=True)
70
+ @click.option(
71
+ "-f",
72
+ "output_format",
73
+ type=click.Choice(["geojson", "geojsonseq", "geoparquet"]),
74
+ required=True,
75
+ )
119
76
  @click.option("-o", "--output", required=False, type=click.Path())
120
77
  @click.option(
121
78
  "-t",
122
79
  "--type",
123
80
  "type_",
124
- type=click.Choice(list(_theme_type_mapping.keys())),
81
+ type=click.Choice(get_all_overture_types()),
125
82
  required=True,
126
83
  )
127
84
  def download(bbox, output_format, output, type_):
128
85
  if output is None:
129
86
  output = sys.stdout
130
87
 
131
- path = _dataset_path(type_)
132
- reader = record_batch_reader(path, bbox)
88
+ reader = record_batch_reader(type_, bbox)
133
89
  if reader is None:
134
90
  return
135
91
 
@@ -153,6 +109,7 @@ class BaseGeoJSONWriter:
153
109
  or output stream. Subclasses should implement write_feature()
154
110
  and finalize() if needed
155
111
  """
112
+
156
113
  def __init__(self, where):
157
114
  self.file_handle = None
158
115
  if isinstance(where, str):
@@ -0,0 +1,102 @@
1
+ from typing import List, Optional
2
+
3
+ import pyarrow as pa
4
+ import pyarrow.compute as pc
5
+ import pyarrow.dataset as ds
6
+ import pyarrow.fs as fs
7
+
8
+
9
+ def record_batch_reader(overture_type, bbox=None) -> Optional[pa.RecordBatchReader]:
10
+ """
11
+ Return a pyarrow RecordBatchReader for the desired bounding box and s3 path
12
+ """
13
+ path = _dataset_path(overture_type)
14
+
15
+ if bbox:
16
+ xmin, ymin, xmax, ymax = bbox
17
+ filter = (
18
+ (pc.field("bbox", "xmin") < xmax)
19
+ & (pc.field("bbox", "xmax") > xmin)
20
+ & (pc.field("bbox", "ymin") < ymax)
21
+ & (pc.field("bbox", "ymax") > ymin)
22
+ )
23
+ else:
24
+ filter = None
25
+
26
+ dataset = ds.dataset(
27
+ path, filesystem=fs.S3FileSystem(anonymous=True, region="us-west-2")
28
+ )
29
+ batches = dataset.to_batches(filter=filter)
30
+
31
+ # to_batches() can yield many batches with no rows. I've seen
32
+ # this cause downstream crashes or other negative effects. For
33
+ # example, the ParquetWriter will emit an empty row group for
34
+ # each one bloating the size of a parquet file. Just omit
35
+ # them so the RecordBatchReader only has non-empty ones. Use
36
+ # the generator syntax so the batches are streamed out
37
+ non_empty_batches = (b for b in batches if b.num_rows > 0)
38
+
39
+ geoarrow_schema = geoarrow_schema_adapter(dataset.schema)
40
+ reader = pa.RecordBatchReader.from_batches(geoarrow_schema, non_empty_batches)
41
+ return reader
42
+
43
+
44
+ def geoarrow_schema_adapter(schema: pa.Schema) -> pa.Schema:
45
+ """
46
+ Convert a geoarrow-compatible schema to a proper geoarrow schema
47
+
48
+ This assumes there is a single "geometry" column with WKB formatting
49
+
50
+ Parameters
51
+ ----------
52
+ schema: pa.Schema
53
+
54
+ Returns
55
+ -------
56
+ pa.Schema
57
+ A copy of the input schema with the geometry field replaced with
58
+ a new one with the proper geoarrow ARROW:extension metadata
59
+
60
+ """
61
+ geometry_field_index = schema.get_field_index("geometry")
62
+ geometry_field = schema.field(geometry_field_index)
63
+ geoarrow_geometry_field = geometry_field.with_metadata(
64
+ {b"ARROW:extension:name": b"geoarrow.wkb"}
65
+ )
66
+
67
+ geoarrow_schema = schema.set(geometry_field_index, geoarrow_geometry_field)
68
+
69
+ return geoarrow_schema
70
+
71
+
72
+ type_theme_map = {
73
+ "locality": "admins",
74
+ "locality_area": "admins",
75
+ "administrative_boundary": "admins",
76
+ "building": "buildings",
77
+ "building_part": "buildings",
78
+ "place": "places",
79
+ "segment": "transportation",
80
+ "connector": "transportation",
81
+ "infrastructure": "base",
82
+ "land": "base",
83
+ "land_use": "base",
84
+ "water": "base",
85
+ }
86
+
87
+
88
+ def _dataset_path(overture_type: str) -> str:
89
+ """
90
+ Returns the s3 path of the Overture dataset to use. This assumes overture_type has
91
+ been validated, e.g. by the CLI
92
+
93
+ """
94
+ # Map of sub-partition "type" to parent partition "theme" for forming the
95
+ # complete s3 path. Could be discovered by reading from the top-level s3
96
+ # location but this allows to only read the files in the necessary partition.
97
+ theme = type_theme_map[overture_type]
98
+ return f"overturemaps-us-west-2/release/2024-04-16-beta.0/theme={theme}/type={overture_type}/"
99
+
100
+
101
+ def get_all_overture_types() -> List[str]:
102
+ return list(type_theme_map.keys())
@@ -1,6 +1,6 @@
1
1
  [tool.poetry]
2
2
  name = "overturemaps"
3
- version = "0.2.0"
3
+ version = "0.3.0"
4
4
  description = "Python tools for interacting with Overture Maps (overturemaps.org) data."
5
5
  authors = ["Jacob Wasserman <jwasserman@meta.com>"]
6
6
  license = "MIT"
@@ -1,55 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: overturemaps
3
- Version: 0.2.0
4
- Summary: Python tools for interacting with Overture Maps (overturemaps.org) data.
5
- License: MIT
6
- Author: Jacob Wasserman
7
- Author-email: jwasserman@meta.com
8
- Requires-Python: >=3.11,<4.0
9
- Classifier: License :: OSI Approved :: MIT License
10
- Classifier: Programming Language :: Python :: 3
11
- Classifier: Programming Language :: Python :: 3.11
12
- Classifier: Programming Language :: Python :: 3.12
13
- Requires-Dist: click (>=8.1.7,<9.0.0)
14
- Requires-Dist: pyarrow (>=15.0.2,<16.0.0)
15
- Requires-Dist: shapely (>=2.0.3,<3.0.0)
16
- Description-Content-Type: text/markdown
17
-
18
- # overturemaps-py
19
-
20
- Official Python command-line tool of the [Overture Maps Foundation](overturemaps.org)
21
-
22
- Note: This repository and project are experimental. Things are likely change including the user interface
23
- until a stable release, but we will keep the documentation here up-to-date.
24
-
25
- ## Basic Usage
26
-
27
- Quick Overview:
28
-
29
- ```
30
- $ overturemaps download --bbox=-71.068,42.353,-71.058,42.363 -o boston.geojson -f geojson -t building
31
- ```
32
-
33
- This command will download the building footprints in the specific bounding box
34
- as GeoJSON and save to a file named "boston.geojson"
35
-
36
- #### `download`
37
- There is currently one option to the `overturemaps` utility, `download`.
38
-
39
- The `download` command option can download Overture Maps data into a few different file formats and
40
- from an optional bounding box. The data is streamed out as it is read and can handle arbitrarily
41
- large bounding boxes or the entire datset.
42
-
43
- Command-line options:
44
- * `bbox` (optional): west, south, east, north longitude and latitude coordinates. When omitted the
45
- entire dataset for the specified type will be downloaded
46
- * `--output`/`-o` (optional): Location of output file. When omitted output will be written to stdout.
47
- * `-f` (required: one of "geojson", "geojsonseq", "geoparquet"): output format
48
- * `--type`/`-t` (required): The Overture map data type to be downloaded. Examples of types are `building`
49
- for building footprints, `place` for POI places data, etc. Run `overturemaps download --help` for the
50
- complete list of allowed types
51
-
52
- ## Installation
53
-
54
- `pip install overturemaps`
55
-
@@ -1,37 +0,0 @@
1
- # overturemaps-py
2
-
3
- Official Python command-line tool of the [Overture Maps Foundation](overturemaps.org)
4
-
5
- Note: This repository and project are experimental. Things are likely change including the user interface
6
- until a stable release, but we will keep the documentation here up-to-date.
7
-
8
- ## Basic Usage
9
-
10
- Quick Overview:
11
-
12
- ```
13
- $ overturemaps download --bbox=-71.068,42.353,-71.058,42.363 -o boston.geojson -f geojson -t building
14
- ```
15
-
16
- This command will download the building footprints in the specific bounding box
17
- as GeoJSON and save to a file named "boston.geojson"
18
-
19
- #### `download`
20
- There is currently one option to the `overturemaps` utility, `download`.
21
-
22
- The `download` command option can download Overture Maps data into a few different file formats and
23
- from an optional bounding box. The data is streamed out as it is read and can handle arbitrarily
24
- large bounding boxes or the entire datset.
25
-
26
- Command-line options:
27
- * `bbox` (optional): west, south, east, north longitude and latitude coordinates. When omitted the
28
- entire dataset for the specified type will be downloaded
29
- * `--output`/`-o` (optional): Location of output file. When omitted output will be written to stdout.
30
- * `-f` (required: one of "geojson", "geojsonseq", "geoparquet"): output format
31
- * `--type`/`-t` (required): The Overture map data type to be downloaded. Examples of types are `building`
32
- for building footprints, `place` for POI places data, etc. Run `overturemaps download --help` for the
33
- complete list of allowed types
34
-
35
- ## Installation
36
-
37
- `pip install overturemaps`
File without changes