@urban-toolkit/autk-grammar 5.0.0 → 5.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -35,15 +35,18 @@ Each entry of `data` loads or builds one table, in order. Its `type` selects its
35
35
 
36
36
  | `type` | Fields | Table |
37
37
  |---|---|---|
38
- | `osm` | `queryArea`, `pbfFileUrl`, `autoLoadLayers` | OpenStreetMap data for an area, from Overpass or a `.osm.pbf` extract (see [OpenStreetMap areas](#openstreetmap-areas)). `autoLoadLayers.layers` splits it into one table per layer, named `<outputTableName>_<layer>`. |
38
+ | `osm` | `queryArea`, `pbfFileUrl`, `autoLoadLayers`, `tagSets` | OpenStreetMap data for an area, from Overpass or a `.osm.pbf` extract (see [OpenStreetMap areas](#openstreetmap-areas)). `autoLoadLayers.layers` splits it into one table per layer, named `<outputTableName>_<layer>`, and [`tagSets`](#tagsets) adds a table for each set of features chosen by their tags. |
39
39
  | `csv` | `csvFileUrl` or `csvObject`, `delimiter`, `geometryColumns` | The rows of a CSV file, or of arrays whose first row names the columns. |
40
40
  | `json` | `jsonFileUrl` or `jsonObject`, `geometryColumns` | The rows of a JSON array. |
41
41
  | `geojson` | `geojsonFileUrl` or `geojsonObject`, `coordinateFormat`, `layerType` | A FeatureCollection. `layerType` sets how a map draws it, for example `polylines`, or `buildings` for polygons; without it, autk-db reads it from the first feature with a geometry. A feature whose geometry is null is a row without one: a map draws nothing for it, and a plot counts it. |
42
42
  | `geotiff` | `geotiffFileUrl`, `coordinateFormat`, `maxRasterCells`, `resampleMethod` | A raster read from a GeoTIFF file. |
43
43
  | `heatmap` | `joinDataRef`, `near`, `grid`, `groupBy` | A raster grid over the workspace's extent, whose cells aggregate the rows of another table. |
44
44
  | `join` | `dataRef`, `joinDataRef`, `near`, `groupBy` | No new table: it joins the rows of `joinDataRef` onto the features of `dataRef`, in place. |
45
+ | `sql` | `query`, `layerType` | The rows of a SQL query over the workspace's tables (see [`sql` sources](#sql-sources)). |
46
+ | `update` | `dataRef`, `strategy`, `idColumn`, `geojsonObject` or `jsonObject` | No new table: it replaces or changes the rows of `dataRef` (see [`update` sources](#update-sources)). |
47
+ | `drop` | `dataRef` | No new table: it drops `dataRef` (see [`drop` sources](#drop-sources)). |
45
48
 
46
- Every source but `join` names its table with `outputTableName`, and every source takes a `workspace` (see [Workspaces](#workspaces)).
49
+ Every source but `join`, `update` and `drop` names its table with `outputTableName`, and every source takes a `workspace` (see [Workspaces](#workspaces)).
47
50
 
48
51
  A `geotiff` source:
49
52
 
@@ -78,12 +81,105 @@ await grammar.run({
78
81
  });
79
82
  ```
80
83
 
84
+ ### Heatmap `normalize`
85
+
86
+ An entry of a heatmap's `groupBy` with `normalize: true` asks for its band's values rescaled from 0 to 1 across the grid's cells, as a join's `normalize` rescales a value across the features. It needs an autk-db with urban-toolkit/autark#132, which also adds `_norm` to the band's label, so a layer names it as `<aggregateFn>.<joinDataRef>_norm`. autk-db 4.1.1 builds the band from the values as they are, under the label it would have without `normalize`. A layer can also name a band by its id, `band_1`, `band_2` and so on, one per entry of `groupBy`, in order.
87
+
88
+ ```ts
89
+ { type: 'heatmap', outputTableName: 'heat', joinDataRef: 'noise', near: { distance: 1000 }, grid: { rows: 20, columns: 20 },
90
+ groupBy: [{ column: 'Unique Key', aggregateFn: 'count', normalize: true }] }
91
+ ```
92
+
93
+ ### `sql` sources
94
+
95
+ A `sql` source saves the rows of a query over the tables of its workspace as a new table, `outputTableName`:
96
+
97
+ | Field | Effect |
98
+ |---|---|
99
+ | `query` | A `SELECT`, or a `WITH` query that ends in one, in DuckDB's SQL, naming the workspace's tables by their names. autk-db refuses any other statement, and any query in which `insert`, `update`, `delete`, `create`, `alter`, `drop`, `truncate` or `replace` appears as a word, even inside a string; the run's message names the source. |
100
+ | `layerType` | How a map draws the new table. The query then returns its geometry in a column named `geometry`, as the workspace's tables hold it, in the workspace's CRS. Without it, the table has no layer type: a plot draws its rows, but a map does not draw it and `grammar.data` does not list it. |
101
+
102
+ A GeoJSON table keeps each feature's properties in a JSON column, `properties`, which `properties->>'name'` reads as text; a CSV or JSON table keeps a column per field. A query that returns `properties` gives each feature those properties; otherwise each column but `geometry` becomes a property. Saving under a table's own name replaces it.
103
+
104
+ ```ts
105
+ await grammar.run({
106
+ data: [
107
+ { type: 'geojson', outputTableName: 'neighborhoods', geojsonFileUrl: 'neighborhoods.geojson' },
108
+ { type: 'csv', outputTableName: 'noise', csvFileUrl: 'noise.csv', geometryColumns: { latColumnName: 'Latitude', longColumnName: 'Longitude' } },
109
+ { type: 'join', dataRef: 'neighborhoods', joinDataRef: 'noise', groupBy: [{ column: 'Unique Key', aggregateFn: 'count' }] },
110
+ {
111
+ type: 'sql',
112
+ outputTableName: 'noisiest',
113
+ query: "SELECT properties->>'ntaname' AS name, CAST(properties->'sjoin'->'count'->>'noise' AS INTEGER) AS complaints, geometry FROM neighborhoods ORDER BY complaints DESC NULLS LAST LIMIT 10",
114
+ layerType: 'polygons',
115
+ },
116
+ ],
117
+ map: { layerRefs: [{ dataRef: 'neighborhoods', color: '#d9d9d9' }, { dataRef: 'noisiest', getFnv: 'complaints' }] },
118
+ });
119
+ ```
120
+
121
+ ### `update` sources
122
+
123
+ An `update` source replaces or changes the rows of a table already loaded, `dataRef`, with rows given in the spec:
124
+
125
+ | Field | Effect |
126
+ |---|---|
127
+ | `strategy` | `replace` swaps every row of the table for the given rows. `update` changes the rows whose `idColumn` value a given row has: in a table with a geometry, each takes the given row's geometry and properties, and no row is added; in a table without one, the given rows replace them, and those that match no row are added. |
128
+ | `idColumn` | With `update`, the column that matches a given row to the table's rows: a column such as `month`, or a feature property as `properties.<name>`. |
129
+ | `geojsonObject` | The rows of a table with a geometry, as a FeatureCollection. autk-db stores their coordinates as they come, so they are in the workspace's CRS, as `grammar.data` gives a table's features. |
130
+ | `jsonObject` | The rows of a table without a geometry, such as a CSV or JSON table loaded without `geometryColumns`, as an array of objects. |
131
+
132
+ A source gives its rows in `geojsonObject` or in `jsonObject`, not both. The maps and plots after it draw the table's rows as they are now. The run's message names the table when autk-db refuses an update.
133
+
134
+ ```ts
135
+ await grammar.run({
136
+ data: [
137
+ { type: 'csv', outputTableName: 'sales', csvObject: [['month', 'total'], ['Jan', 3], ['Feb', 5]] },
138
+ { type: 'update', dataRef: 'sales', strategy: 'update', idColumn: 'month', jsonObject: [{ month: 'Feb', total: 9 }, { month: 'Mar', total: 4 }] },
139
+ ],
140
+ plot: { dataRef: 'sales', mark: 'bar', axis: ['month', 'total'] }, // Jan 3, Feb 9, Mar 4
141
+ });
142
+ ```
143
+
144
+ ### `drop` sources
145
+
146
+ A `drop` source drops a table, `dataRef`, from its workspace: the sources, compute passes, maps and plots after it no longer find it, and `grammar.data` no longer lists it. It suits a table a spec needs only on the way, such as the rows a join aggregates. The run refuses a drop of a table the workspace does not hold, with a message that names the table.
147
+
148
+ ```ts
149
+ data: [
150
+ { type: 'geojson', outputTableName: 'neighborhoods', geojsonFileUrl: 'neighborhoods.geojson' },
151
+ { type: 'csv', outputTableName: 'noise', csvFileUrl: 'noise.csv', geometryColumns: true },
152
+ { type: 'join', dataRef: 'neighborhoods', joinDataRef: 'noise', groupBy: [{ column: 'Unique Key', aggregateFn: 'count' }] },
153
+ { type: 'drop', dataRef: 'noise' },
154
+ ],
155
+ ```
156
+
81
157
  ### Workspaces
82
158
 
83
159
  A workspace is an isolated set of tables with its own extent. The first source loaded into a workspace sets its extent, and later sources are cut to it, so two areas that must not clip each other go in two workspaces. A table is found by its name inside its workspace, so two workspaces may hold tables of the same name.
84
160
 
85
161
  `workspace` names one, on a source (where its table goes), a compute pass (where it reads and writes), a map or a plot (where it reads its tables). An entry without one uses the default workspace.
86
162
 
163
+ ### `workspaces`
164
+
165
+ A spec's `workspaces` sets how a workspace stores its tables, one entry per workspace, before the first source loads:
166
+
167
+ | Field | Effect |
168
+ |---|---|
169
+ | `name` | The workspace, by the name its entries give as their `workspace`. Without it, the default workspace. |
170
+ | `coordinateFormat` | The CRS the workspace stores its tables in, as an EPSG string such as `EPSG:3857`. Each source is transformed to it as it loads, and its maps draw in its units. autk-db's default is `EPSG:3395`, World Mercator, in meters. |
171
+ | `precisionGrid` | The grid every stored coordinate is rounded to, in the CRS's units, greater than 0. autk-db's default is 0.01, a centimeter in World Mercator. A geometry the grid collapses is refused as it loads. |
172
+
173
+ Both fields are required, and a run refuses two entries for one workspace. A workspace without an entry keeps autk-db's defaults. The lengths given in workspace units are in the CRS's units: a `near.distance`, a layer's `width`, a camera's `zoom`. A layer's `pointSize`, in meters, is converted for World Mercator and Web Mercator, and read as planar units in any other CRS.
174
+
175
+ ```ts
176
+ await grammar.run({
177
+ workspaces: [{ coordinateFormat: 'EPSG:3857', precisionGrid: 1 }], // the default workspace, in Web Mercator to the meter
178
+ data: [{ type: 'geojson', outputTableName: 'parcels', geojsonFileUrl: 'parcels.geojson' }],
179
+ map: { layerRefs: [{ dataRef: 'parcels' }] },
180
+ });
181
+ ```
182
+
87
183
  ## OpenStreetMap areas
88
184
 
89
185
  An `osm` source's `queryArea` names the area to load in one of two ways, never both:
@@ -108,6 +204,58 @@ await grammar.run({
108
204
  });
109
205
  ```
110
206
 
207
+ ### `tagSets`
208
+
209
+ An `osm` source's `tagSets` choose features by their tags. Each set becomes a table of one geometry type, named `<outputTableName>_<name>_<type>` and cut to the source's area. Tag sets are read from the Overpass API only: a source with `pbfFileUrl` cannot take them, and the run refuses one that does, with a message that names the source.
210
+
211
+ | Field | Effect |
212
+ |---|---|
213
+ | `name` | Names the set's table: a lowercase letter, then up to 31 lowercase letters, digits or `_`. No two sets of a source share a name. |
214
+ | `type` | The table's geometry: `points`, the tagged nodes; `polylines`, the open ways and the ways tagged as lines (`highway`, `barrier`, `railway`, `waterway`, or `area=no`); `polygons`, the other closed ways and the multipolygon relations. |
215
+ | `tags` | 1 to 64 tags, each `{ key, value }`: a feature is in the set when it has any one of them. Without `value`, the key with any value matches. |
216
+
217
+ A source takes up to 8 sets, and may leave out `autoLoadLayers` to load its sets alone. A set that matches no feature in the area makes no table. Each feature's properties are its tags.
218
+
219
+ ```ts
220
+ await grammar.run({
221
+ data: [{
222
+ type: 'osm',
223
+ outputTableName: 'downtown',
224
+ queryArea: { bbox: [-74.0125, 40.705, -74.008, 40.7085] },
225
+ tagSets: [
226
+ { name: 'cafes', type: 'points', tags: [{ key: 'amenity', value: 'cafe' }] },
227
+ { name: 'paths', type: 'polylines', tags: [{ key: 'highway', value: 'footway' }, { key: 'highway', value: 'path' }] },
228
+ ],
229
+ }],
230
+ map: { layerRefs: [{ dataRef: 'downtown_paths_polylines' }, { dataRef: 'downtown_cafes_points', color: '#8c564b' }] },
231
+ });
232
+ ```
233
+
234
+ ### `forceRefresh`
235
+
236
+ autk-db keeps the Overpass API's answer for an `osm` source in the browser's cache for 24 hours, by its area, its layers and its tag sets, and a later run of the same source reads it from there. `forceRefresh: true` queries the API again, and caches the new answer. A source with `pbfFileUrl` reads its file each time, and ignores it.
237
+
238
+ ```ts
239
+ { type: 'osm', outputTableName: 'loop', queryArea: { geocodeArea: 'Chicago', areas: ['Loop'] }, autoLoadLayers: { layers: ['roads'] }, forceRefresh: true }
240
+ ```
241
+
242
+ ### `autoLoadLayers.coordinateFormat`
243
+
244
+ The CRS of an `osm` source's coordinates, as an EPSG string. It defaults to `EPSG:4326`, the degrees the Overpass API and OpenStreetMap's extracts give, so a source sets it only for data in another CRS. The layers and the tag sets are read in it, and stored in the workspace's CRS.
245
+
246
+ ```ts
247
+ { type: 'osm', outputTableName: 'area', pbfFileUrl: 'area-nad83.osm.pbf', queryArea: { bbox: [-74.02, 40.70, -74.00, 40.72] }, autoLoadLayers: { layers: ['roads'], coordinateFormat: 'EPSG:4269' } }
248
+ ```
249
+
250
+ ### `osmElements`
251
+
252
+ A map layer or a plot with `osmElements: true` reads an OpenStreetMap layer table as one feature per OSM element it was built from, rather than one per row. Each feature's properties are the element's tags, its `osm_type` (`node`, `way` or `relation`) and its `osm_id`; in a buildings table, whose parts are elements of their own, also the `building_id` of its building. A part takes only its own tags, so a buildings layer draws only the parts whose tags give their height. A table that is not an OSM layer, a surface, and an OSM table an `update` source changed are read row by row, as without it.
253
+
254
+ ```ts
255
+ map: { layerRefs: [{ dataRef: 'downtown_roads', osmElements: true, getFnv: 'highway', colorMapInterpolator: ColorMapInterpolator.CAT_TABLEAU10 }] },
256
+ plot: { dataRef: 'downtown_roads', mark: 'table', axis: ['osm_id', 'highway', 'name'], osmElements: true },
257
+ ```
258
+
111
259
  ## Compute
112
260
 
113
261
  Each entry of `compute` runs on the GPU with autk-compute, in order, once the data is loaded. Its `type` is `gpgpu`, the default, or `render`.
@@ -172,10 +320,10 @@ A map's `camera` moves its view from the one that fits its layers:
172
320
  | Option | Effect |
173
321
  |---|---|
174
322
  | `zoom` | The eye's distance from the point the view looks at: `10000 / 2^(zoom - 10)` workspace units, so each step halves it. Without it, the distance of the view that fits the layers. |
175
- | `pitch` | The tilt from straight down, from 0 to 90 degrees. |
323
+ | `pitch` | The tilt from straight down, in degrees, at least 0 and less than 90. At 90 the eye would sit on the ground, edge-on to the flat layers, and the map could not pan, so a run refuses it. |
176
324
  | `bearing` | The compass heading the view faces, clockwise from north, from 0 to 360 degrees. |
177
325
 
178
- The camera turns around the point the map looks at, which the view that fits the layers sets: the eye moves on a sphere around that point, so the layers stay in view at any pitch and bearing.
326
+ The camera turns around the point the map looks at, which the view that fits the layers sets: the eye moves on a sphere around that point, so the layers stay in view at any pitch and bearing. The view's up direction stays across its line of sight, as autk-map's own tilt leaves it, so the map's pan, tilt and zoom at the cursor carry on from the spec's view as they do from a view the user tilted.
179
327
 
180
328
  ```ts
181
329
  map: { id: 'downtown', camera: { zoom: 15, pitch: 50, bearing: 30 }, layerRefs: [{ dataRef: 'buildings' }] }
@@ -329,11 +477,11 @@ A host that runs part of a spec itself, a server that loads its tables for examp
329
477
 
330
478
  | Export | Effect |
331
479
  |---|---|
332
- | `createDataAdapter(targets?, cache?)` | The data adapter. Its `resolveSource(db, source)` loads one `data` entry into an autk-db database, a new one when `db` is undefined, and returns it. `cache` receives each GeoJSON table and each table a join wrote. |
480
+ | `createDataAdapter(targets?, cache?)` | The data adapter. Its `resolveWorkspaces(db, workspaces)` sets up a spec's `workspaces` before the first source loads, and its `resolveSource(db, source)` loads one `data` entry. Each takes an autk-db database, a new one when `db` is undefined, and returns it. `cache` receives each GeoJSON table and each table a join or an update wrote. |
333
481
  | `createComputeAdapter(cache?, types?)` | The compute adapter. Its `resolveCompute(db, pass)` runs one `compute` entry on the GPU. The results are kept only in `cache`, and `types` receives the layer type of each table a pass wrote under its own `outputTableName`. |
334
482
  | `enterWorkspace(db, workspace?)` | Makes an entry's workspace (the default one without a name) the one autk-db works in, and switches only when another is current. Every adapter calls it before it reads or writes an entry's tables. |
335
483
  | `tableKey(workspace, name)` | The key of a table in those caches: its name in the default workspace, `<workspace>/<name>` in another. |
336
- | `applyCamera(map, camera)` | Moves an `AutkMap`'s camera as a map's `camera` says, from the view that fits its layers. The map adapter calls it once the layers are loaded, before the map's first `draw()`. |
484
+ | `applyCamera(map, camera)` | Moves an `AutkMap`'s camera as a map's `camera` says, from the view that fits its layers, and refuses a `pitch` that is not at least 0 and less than 90. The map adapter calls it once the layers are loaded, before the map's first `draw()`. |
337
485
  | `arrangeViews(spec)` | Where each view of a spec goes on the page, as its `layout` says: a grid of panes, one per view (see [Arranging the views](#arranging-the-views)). It refuses a `layout` it cannot arrange, with a sentence that names the value. |
338
486
 
339
487
  `@urban-toolkit/autk-grammar/data` exports `createDataAdapter`, `enterWorkspace` and `tableKey` with their types, as an ES module and a CommonJS one. Its code imports `@urban-toolkit/autk-db` alone, never autk-map, autk-plot, autk-compute or autk-core, so it loads where they cannot, such as Node:
@@ -342,7 +490,8 @@ A host that runs part of a spec itself, a server that loads its tables for examp
342
490
  import { createDataAdapter } from '@urban-toolkit/autk-grammar/data';
343
491
 
344
492
  const adapter = createDataAdapter();
345
- let db: unknown; // the AutkDb the first source creates
493
+ let db: unknown; // the AutkDb the first call creates
494
+ if (spec.workspaces) db = await adapter.resolveWorkspaces(db, spec.workspaces);
346
495
  for (const source of spec.data ?? []) db = await adapter.resolveSource(db, source);
347
496
  ```
348
497