api-dock 0.7.1__tar.gz → 0.8.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {api_dock-0.7.1 → api_dock-0.8.0}/PKG-INFO +220 -1
- {api_dock-0.7.1 → api_dock-0.8.0}/README.md +219 -0
- api_dock-0.8.0/api_dock/database_config.py +972 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/example_api_dock_config/config.yaml +8 -0
- api_dock-0.8.0/api_dock/example_api_dock_config/databases/config.yaml +82 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/fast_api.py +33 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/flask_api.py +31 -0
- api_dock-0.8.0/api_dock/listings.py +346 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/route_mapper.py +53 -13
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/sql_builder.py +109 -12
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/storage_auth.py +133 -36
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/types.py +59 -2
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/PKG-INFO +220 -1
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/SOURCES.txt +4 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/pyproject.toml +1 -1
- api_dock-0.8.0/tests/test_listings.py +221 -0
- api_dock-0.8.0/tests/test_shared_database_config.py +835 -0
- api_dock-0.7.1/api_dock/database_config.py +0 -441
- {api_dock-0.7.1 → api_dock-0.8.0}/LICENSE.md +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/__init__.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/auth.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/cli.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/config.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/config_discovery.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/encryption.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/example_api_dock_config/databases/example_db.yaml +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock/example_api_dock_config/remotes/example_remote.yaml +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/dependency_links.txt +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/entry_points.txt +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/requires.txt +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/api_dock.egg-info/top_level.txt +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/setup.cfg +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/tests/test_inject_cookies.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/tests/test_proxy_pipeline.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/tests/test_sql_builder.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/tests/test_sql_selector.py +0 -0
- {api_dock-0.7.1 → api_dock-0.8.0}/tests/test_types.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: api_dock
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.8.0
|
|
4
4
|
Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
|
|
5
5
|
Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
|
|
6
6
|
License-Expression: BSD-3-Clause
|
|
@@ -95,6 +95,7 @@ Here is an example:
|
|
|
95
95
|
api_dock_config
|
|
96
96
|
├── config.yaml # The default main-config file
|
|
97
97
|
├── databases
|
|
98
|
+
│ ├── config.yaml # (optional) shared tables/schemas for all databases
|
|
98
99
|
│ ├── unversioned_db.yaml # Database config without versioning
|
|
99
100
|
│ └── versioned_db # Folder containing database configs for different versions
|
|
100
101
|
│ ├── 0.1.yaml
|
|
@@ -231,6 +232,58 @@ The optional `settings` section controls HTTP behavior:
|
|
|
231
232
|
|
|
232
233
|
- **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
|
|
233
234
|
|
|
235
|
+
### Catalog Endpoints (`expose`)
|
|
236
|
+
|
|
237
|
+
The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
|
|
238
|
+
|
|
239
|
+
```yaml
|
|
240
|
+
# Enable all three defaults: /databases, /remotes, /sources
|
|
241
|
+
expose: true
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
```yaml
|
|
245
|
+
GET /databases
|
|
246
|
+
# [{"model": "birdnet", "version": "2.4"},
|
|
247
|
+
# {"model": "birdnet", "version": "3.0"},
|
|
248
|
+
# {"model": "owl", "version": "0.5"}]
|
|
249
|
+
|
|
250
|
+
GET /sources # databases + remotes, combined
|
|
251
|
+
# [{"model": "birdnet", "version": "2.4"}, ..., {"model": "core", "version": "0.5.0"}]
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Versions are the config filename stems, so semver like `0.5.0` is preserved; unversioned sources report `version: null`. Enable only what you want, and control the output shape with `dict`:
|
|
255
|
+
|
|
256
|
+
```yaml
|
|
257
|
+
expose:
|
|
258
|
+
dict: false # return "model/version" strings instead of {model, version} dicts
|
|
259
|
+
databases: true # add /databases
|
|
260
|
+
remotes: true # add /remotes
|
|
261
|
+
# sources omitted → not added
|
|
262
|
+
|
|
263
|
+
# GET /databases -> ["birdnet/2.4", "birdnet/3.0", "owl/0.5"]
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
Each of `databases` / `remotes` / `sources` accepts several forms:
|
|
267
|
+
|
|
268
|
+
```yaml
|
|
269
|
+
expose:
|
|
270
|
+
databases: false # do not add the endpoint
|
|
271
|
+
|
|
272
|
+
remotes: "list/remotes" # custom route path (same as true, but at /list/remotes)
|
|
273
|
+
|
|
274
|
+
databases: # explicit list of models (optionally version-filtered)
|
|
275
|
+
- birdnet
|
|
276
|
+
- owl:
|
|
277
|
+
versions: [4.0, 5.0] # ints/floats/strings all match the "4.0"/"5.0" stems
|
|
278
|
+
|
|
279
|
+
sources: # most explicit form
|
|
280
|
+
route: "list/sources"
|
|
281
|
+
include: [birdnet, core] # true (default) | false | list of models
|
|
282
|
+
dict: true # per-endpoint override of the top-level `dict`
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Each listing route is served both with and without a trailing slash (e.g. `/sources` and `/sources/` both work), so it doesn't matter which convention your clients use. Custom multi-segment routes (e.g. `list/databases`) take precedence over the `/{remote}/{path}` proxy. If a listing route would shadow a configured remote/database, or an `include` names something that doesn't exist, API Dock emits a startup warning. The exposed routes are also reflected in the root (`/`) metadata's `endpoints`.
|
|
286
|
+
|
|
234
287
|
---
|
|
235
288
|
|
|
236
289
|
## Remote Configurations
|
|
@@ -356,6 +409,7 @@ Database configurations are stored in `api_dock_config/databases/` directory. Ea
|
|
|
356
409
|
- **tables**: Mapping of table names to file paths (supports S3, GCS, HTTPS, local paths)
|
|
357
410
|
- **queries**: Named SQL queries for reuse
|
|
358
411
|
- **routes**: REST endpoints mapped to SQL queries
|
|
412
|
+
- **schema** (optional): the shared schema (from `databases/config.yaml`) this config's `[[table]]` references fall back to. See [Shared Tables and Schemas](#shared-tables-and-schemas-databasesconfigyaml)
|
|
359
413
|
|
|
360
414
|
### Syntax
|
|
361
415
|
|
|
@@ -433,6 +487,171 @@ routes:
|
|
|
433
487
|
sql: "[[get_permissions]]"
|
|
434
488
|
```
|
|
435
489
|
|
|
490
|
+
### Shared Tables and Schemas (`databases/config.yaml`)
|
|
491
|
+
|
|
492
|
+
When several databases or versions read the same tables, define them once in the optional `api_dock_config/databases/config.yaml`. Everything lives under a `database` key: `meta` holds default table metadata, `schema` holds named groups of tables, and every other key is a global table.
|
|
493
|
+
|
|
494
|
+
```yaml
|
|
495
|
+
# api_dock_config/databases/config.yaml
|
|
496
|
+
database:
|
|
497
|
+
# global tables, available as [[table1]] in any database config
|
|
498
|
+
table1:
|
|
499
|
+
uri: s3://your-bucket/table1.parquet
|
|
500
|
+
table3:
|
|
501
|
+
uri: s3://your-other-bucket/table3.parquet
|
|
502
|
+
region: us-west-1 # a table's own keys override `meta`
|
|
503
|
+
public: false
|
|
504
|
+
|
|
505
|
+
# defaults applied to every table (shared tables and the tables in each version config)
|
|
506
|
+
meta:
|
|
507
|
+
region: us-west-2
|
|
508
|
+
public: true
|
|
509
|
+
|
|
510
|
+
# schemas, available as [[birdnet_2p4.detections]] in any route
|
|
511
|
+
schema:
|
|
512
|
+
birdnet_2p4:
|
|
513
|
+
detections:
|
|
514
|
+
uri: s3://your-bucket/birdnet/2.4/detections.parquet
|
|
515
|
+
birdnet_3p0:
|
|
516
|
+
detections:
|
|
517
|
+
uri: s3://your-bucket/birdnet/3.0/detections.parquet
|
|
518
|
+
public: false
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
A version config can name the schema it uses with `schema:`. An unqualified `[[name]]` is then looked up in order, first match wins:
|
|
522
|
+
|
|
523
|
+
1. the version config's own `tables`
|
|
524
|
+
2. its `schema:` in the shared config
|
|
525
|
+
3. the shared config's global tables
|
|
526
|
+
|
|
527
|
+
```yaml
|
|
528
|
+
# api_dock_config/databases/birdnet/2.4.yaml
|
|
529
|
+
name: birdnet
|
|
530
|
+
schema: birdnet_2p4
|
|
531
|
+
tables:
|
|
532
|
+
revisions: s3://your-bucket/birdnet/2.4/revisions.parquet # local to this version
|
|
533
|
+
|
|
534
|
+
routes:
|
|
535
|
+
# [[detections]] isn't in `tables`, so it comes from the birdnet_2p4 schema
|
|
536
|
+
- route: recordings/{{recording_id}}/detections
|
|
537
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
Any route can reference any schema directly as `[[schema.table]]`, so a different database (e.g. `owl/5.0`) can query `[[birdnet_2p4.detections]]`. To keep a table private to one database/version, define it in that version's `tables` instead. Qualified references are exposed to DuckDB as real views, so you can also use the full name or your own alias in plain SQL:
|
|
541
|
+
|
|
542
|
+
```yaml
|
|
543
|
+
- route: detections/
|
|
544
|
+
sql: >
|
|
545
|
+
SELECT detections.common_name, COUNT(revisions.id) AS revcount
|
|
546
|
+
FROM [[birdnet_2p4.detections]]
|
|
547
|
+
LEFT JOIN [[revisions]] ON revisions.observation_id = birdnet_2p4.detections.id
|
|
548
|
+
GROUP BY birdnet_2p4.detections.common_name
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
expands to
|
|
552
|
+
|
|
553
|
+
```sql
|
|
554
|
+
SELECT detections.common_name, COUNT(revisions.id) AS revcount
|
|
555
|
+
FROM birdnet_2p4.detections
|
|
556
|
+
LEFT JOIN 's3://your-bucket/birdnet/2.4/revisions.parquet' AS revisions ON revisions.observation_id = birdnet_2p4.detections.id
|
|
557
|
+
GROUP BY birdnet_2p4.detections.common_name
|
|
558
|
+
```
|
|
559
|
+
|
|
560
|
+
Notes:
|
|
561
|
+
- After `FROM`/`JOIN`, `[[schema.table]]` becomes `schema.table` with no alias, so `FROM [[birdnet_2p4.detections]] o` works. Elsewhere it becomes the bare table name (`detections`), because DuckDB doesn't accept `schema.table.*`.
|
|
562
|
+
- Schema and table names used as `[[schema.table]]` must be plain identifiers (letters, digits, underscores).
|
|
563
|
+
- Storage credentials are set per table. Tables whose `region`/`public` differ from the rest get their own S3 secret scoped to their path, so one query can mix regions and public/private buckets.
|
|
564
|
+
- Views are created only for the `[[schema.table]]` tables a query actually references.
|
|
565
|
+
|
|
566
|
+
#### Inline database configs (`slugs`)
|
|
567
|
+
|
|
568
|
+
Simple database/version configs (often just a description and a `schema`) can live in the shared file instead of in their own files. Config files keep working, and the two can be mixed, even for the same database:
|
|
569
|
+
|
|
570
|
+
```yaml
|
|
571
|
+
# api_dock_config/databases/config.yaml
|
|
572
|
+
slugs:
|
|
573
|
+
- name: birdnet-bullfrog # the database slug in the URL
|
|
574
|
+
version: "2.5" # one version...
|
|
575
|
+
description: American Bullfrog Classifier from Birdnet 2.4
|
|
576
|
+
schema: birdnet_bullfrog_2p5v0p5
|
|
577
|
+
- name: birdnet-apple
|
|
578
|
+
authors: [API Team] # ...or several; keys here are defaults for each version
|
|
579
|
+
versions:
|
|
580
|
+
- version: "1.0"
|
|
581
|
+
description: Apple Classifier 1.0
|
|
582
|
+
schema: birdnet_apple_1p0
|
|
583
|
+
- version: "12.0"
|
|
584
|
+
description: Apple Classifier 12.0
|
|
585
|
+
schema: birdnet_apple_12p0
|
|
586
|
+
- name: notes # no version/versions = an unversioned database
|
|
587
|
+
tables:
|
|
588
|
+
notes: s3://your-bucket/notes.parquet
|
|
589
|
+
```
|
|
590
|
+
|
|
591
|
+
- Each entry (or each `versions` item) takes the same keys as a database config file: `description`, `authors`, `schema`, `tables`, `routes`, `query_params`, and so on. Shared `routes`/`query_params` (below), including `include`/`exclude`, apply to them like any other database/version.
|
|
592
|
+
- Like file-based databases, a slug is only served if it's listed under `databases:` in the main `config.yaml`.
|
|
593
|
+
- A database's versions are the union of its version files and its `slugs` versions, so `latest`, the `/{database}` versions listing, and the `expose` catalog endpoints all see both. If a file and a slug define the same database/version, the file wins.
|
|
594
|
+
- Quote versions (`version: "2.10"`). Unquoted YAML numbers are floats, so `2.10` would become `"2.1"`.
|
|
595
|
+
- A malformed `slugs` section (missing `name`, both `version` and `versions`, a duplicate version, or a mix of versioned and unversioned entries for one name) returns a 500 "Shared database configuration error".
|
|
596
|
+
|
|
597
|
+
#### Shared routes and query params
|
|
598
|
+
|
|
599
|
+
The shared file can also define top-level `routes` and `query_params`. These are added to **every** database/version, which is handy when each model/version serves the same endpoints over its own `schema`:
|
|
600
|
+
|
|
601
|
+
```yaml
|
|
602
|
+
# api_dock_config/databases/config.yaml
|
|
603
|
+
database:
|
|
604
|
+
...
|
|
605
|
+
|
|
606
|
+
routes:
|
|
607
|
+
- route: recordings/{{recording_id}}/detections/
|
|
608
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
|
|
609
|
+
- route: detections/{{id}}
|
|
610
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].id = {{id}}
|
|
611
|
+
- route: not_for_everyone/{{id}}
|
|
612
|
+
sql: SELECT [[other]].* FROM [[other]] WHERE [[other]].id = {{id}}
|
|
613
|
+
exclude: # don't add this route to these slug/versions
|
|
614
|
+
- 'slug1/3.0'
|
|
615
|
+
- slug: slug2
|
|
616
|
+
version: 2.3
|
|
617
|
+
- slug: slug3
|
|
618
|
+
version: '*' # '*' = every version
|
|
619
|
+
|
|
620
|
+
# the same route defined twice: one for everything except birdnet/2.4, one only for it
|
|
621
|
+
- route: detections/
|
|
622
|
+
exclude: ['birdnet/2.4']
|
|
623
|
+
sql: SELECT [[detections]].* FROM [[detections]]
|
|
624
|
+
- route: detections/
|
|
625
|
+
include: ['birdnet/2.4'] # ONLY add this route to these slug/versions
|
|
626
|
+
sql: SELECT [[detections]].*, [[revisions]].id AS revision_id FROM [[detections]] LEFT JOIN [[revisions]] ON [[revisions]].observation_id = [[detections]].id
|
|
627
|
+
|
|
628
|
+
query_params:
|
|
629
|
+
- confidence:
|
|
630
|
+
sql: "[[detections]].confidence >= {{confidence}}"
|
|
631
|
+
- start_time:
|
|
632
|
+
sql: "[[detections]].start_time >= {{start_time}}"
|
|
633
|
+
exclude: ['slug1/3.9']
|
|
634
|
+
- limit:
|
|
635
|
+
sql_append: LIMIT {{limit}}
|
|
636
|
+
|
|
637
|
+
# limit ALL shared routes / query params to these slug/versions
|
|
638
|
+
route_inclusions: ['birdnet', 'owl/5.0']
|
|
639
|
+
query_inclusions: [] # empty or missing = no restriction
|
|
640
|
+
|
|
641
|
+
# opt slug/versions out of ALL shared routes / query params
|
|
642
|
+
route_exclusions: ['legacy_db']
|
|
643
|
+
query_exclusions:
|
|
644
|
+
- slug: slug4
|
|
645
|
+
version: 1.0
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
Rules:
|
|
649
|
+
- **The version config wins.** Its own routes come first and replace any shared route with the same shape. Shape means the same path segments; `{{param}}` names and leading/trailing slashes are ignored, so `detections/{{id}}` and `/detections/{{detection_id}}/` are the same route. Routes the version config adds on top are kept.
|
|
650
|
+
- Shared `query_params` behave like a version config's top-level `query_params`. They apply to every route, and a param with the same name in the version config (or on a route) overrides the shared one.
|
|
651
|
+
- `include` and `route_inclusions`/`query_inclusions` are the opposite of `exclude` and `route_exclusions`/`query_exclusions`. When given (non-empty), the route or query param is added **only** to the listed slug/versions. A shared item is added only if it passes both the top-level lists and its own `include`/`exclude`.
|
|
652
|
+
- The same route (by shape) or query param (by name) can appear more than once in the shared file. Each database/version gets the first one whose `include`/`exclude` select it, so complementary `include`/`exclude` lists give different databases different versions of an endpoint.
|
|
653
|
+
- `include`, `exclude` and the four top-level lists take a list of `'<slug>/<version>'` strings or `{slug: <slug>, version: <version>}` mappings. `'<slug>'`, `'<slug>/*'` or a missing/`'*'` version match every version, including unversioned databases. Versions compare numerically when possible (`2.3`, `"2.3"`), and `latest` is resolved before matching.
|
|
654
|
+
|
|
436
655
|
**For more details**, see the [SQL Database Support Wiki](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support).
|
|
437
656
|
|
|
438
657
|
---
|
|
@@ -56,6 +56,7 @@ Here is an example:
|
|
|
56
56
|
api_dock_config
|
|
57
57
|
├── config.yaml # The default main-config file
|
|
58
58
|
├── databases
|
|
59
|
+
│ ├── config.yaml # (optional) shared tables/schemas for all databases
|
|
59
60
|
│ ├── unversioned_db.yaml # Database config without versioning
|
|
60
61
|
│ └── versioned_db # Folder containing database configs for different versions
|
|
61
62
|
│ ├── 0.1.yaml
|
|
@@ -192,6 +193,58 @@ The optional `settings` section controls HTTP behavior:
|
|
|
192
193
|
|
|
193
194
|
- **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
|
|
194
195
|
|
|
196
|
+
### Catalog Endpoints (`expose`)
|
|
197
|
+
|
|
198
|
+
The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
|
|
199
|
+
|
|
200
|
+
```yaml
|
|
201
|
+
# Enable all three defaults: /databases, /remotes, /sources
|
|
202
|
+
expose: true
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
```yaml
|
|
206
|
+
GET /databases
|
|
207
|
+
# [{"model": "birdnet", "version": "2.4"},
|
|
208
|
+
# {"model": "birdnet", "version": "3.0"},
|
|
209
|
+
# {"model": "owl", "version": "0.5"}]
|
|
210
|
+
|
|
211
|
+
GET /sources # databases + remotes, combined
|
|
212
|
+
# [{"model": "birdnet", "version": "2.4"}, ..., {"model": "core", "version": "0.5.0"}]
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
Versions are the config filename stems, so semver like `0.5.0` is preserved; unversioned sources report `version: null`. Enable only what you want, and control the output shape with `dict`:
|
|
216
|
+
|
|
217
|
+
```yaml
|
|
218
|
+
expose:
|
|
219
|
+
dict: false # return "model/version" strings instead of {model, version} dicts
|
|
220
|
+
databases: true # add /databases
|
|
221
|
+
remotes: true # add /remotes
|
|
222
|
+
# sources omitted → not added
|
|
223
|
+
|
|
224
|
+
# GET /databases -> ["birdnet/2.4", "birdnet/3.0", "owl/0.5"]
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Each of `databases` / `remotes` / `sources` accepts several forms:
|
|
228
|
+
|
|
229
|
+
```yaml
|
|
230
|
+
expose:
|
|
231
|
+
databases: false # do not add the endpoint
|
|
232
|
+
|
|
233
|
+
remotes: "list/remotes" # custom route path (same as true, but at /list/remotes)
|
|
234
|
+
|
|
235
|
+
databases: # explicit list of models (optionally version-filtered)
|
|
236
|
+
- birdnet
|
|
237
|
+
- owl:
|
|
238
|
+
versions: [4.0, 5.0] # ints/floats/strings all match the "4.0"/"5.0" stems
|
|
239
|
+
|
|
240
|
+
sources: # most explicit form
|
|
241
|
+
route: "list/sources"
|
|
242
|
+
include: [birdnet, core] # true (default) | false | list of models
|
|
243
|
+
dict: true # per-endpoint override of the top-level `dict`
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
Each listing route is served both with and without a trailing slash (e.g. `/sources` and `/sources/` both work), so it doesn't matter which convention your clients use. Custom multi-segment routes (e.g. `list/databases`) take precedence over the `/{remote}/{path}` proxy. If a listing route would shadow a configured remote/database, or an `include` names something that doesn't exist, API Dock emits a startup warning. The exposed routes are also reflected in the root (`/`) metadata's `endpoints`.
|
|
247
|
+
|
|
195
248
|
---
|
|
196
249
|
|
|
197
250
|
## Remote Configurations
|
|
@@ -317,6 +370,7 @@ Database configurations are stored in `api_dock_config/databases/` directory. Ea
|
|
|
317
370
|
- **tables**: Mapping of table names to file paths (supports S3, GCS, HTTPS, local paths)
|
|
318
371
|
- **queries**: Named SQL queries for reuse
|
|
319
372
|
- **routes**: REST endpoints mapped to SQL queries
|
|
373
|
+
- **schema** (optional): the shared schema (from `databases/config.yaml`) this config's `[[table]]` references fall back to. See [Shared Tables and Schemas](#shared-tables-and-schemas-databasesconfigyaml)
|
|
320
374
|
|
|
321
375
|
### Syntax
|
|
322
376
|
|
|
@@ -394,6 +448,171 @@ routes:
|
|
|
394
448
|
sql: "[[get_permissions]]"
|
|
395
449
|
```
|
|
396
450
|
|
|
451
|
+
### Shared Tables and Schemas (`databases/config.yaml`)
|
|
452
|
+
|
|
453
|
+
When several databases or versions read the same tables, define them once in the optional `api_dock_config/databases/config.yaml`. Everything lives under a `database` key: `meta` holds default table metadata, `schema` holds named groups of tables, and every other key is a global table.
|
|
454
|
+
|
|
455
|
+
```yaml
|
|
456
|
+
# api_dock_config/databases/config.yaml
|
|
457
|
+
database:
|
|
458
|
+
# global tables, available as [[table1]] in any database config
|
|
459
|
+
table1:
|
|
460
|
+
uri: s3://your-bucket/table1.parquet
|
|
461
|
+
table3:
|
|
462
|
+
uri: s3://your-other-bucket/table3.parquet
|
|
463
|
+
region: us-west-1 # a table's own keys override `meta`
|
|
464
|
+
public: false
|
|
465
|
+
|
|
466
|
+
# defaults applied to every table (shared tables and the tables in each version config)
|
|
467
|
+
meta:
|
|
468
|
+
region: us-west-2
|
|
469
|
+
public: true
|
|
470
|
+
|
|
471
|
+
# schemas, available as [[birdnet_2p4.detections]] in any route
|
|
472
|
+
schema:
|
|
473
|
+
birdnet_2p4:
|
|
474
|
+
detections:
|
|
475
|
+
uri: s3://your-bucket/birdnet/2.4/detections.parquet
|
|
476
|
+
birdnet_3p0:
|
|
477
|
+
detections:
|
|
478
|
+
uri: s3://your-bucket/birdnet/3.0/detections.parquet
|
|
479
|
+
public: false
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
A version config can name the schema it uses with `schema:`. An unqualified `[[name]]` is then looked up in order, first match wins:
|
|
483
|
+
|
|
484
|
+
1. the version config's own `tables`
|
|
485
|
+
2. its `schema:` in the shared config
|
|
486
|
+
3. the shared config's global tables
|
|
487
|
+
|
|
488
|
+
```yaml
|
|
489
|
+
# api_dock_config/databases/birdnet/2.4.yaml
|
|
490
|
+
name: birdnet
|
|
491
|
+
schema: birdnet_2p4
|
|
492
|
+
tables:
|
|
493
|
+
revisions: s3://your-bucket/birdnet/2.4/revisions.parquet # local to this version
|
|
494
|
+
|
|
495
|
+
routes:
|
|
496
|
+
# [[detections]] isn't in `tables`, so it comes from the birdnet_2p4 schema
|
|
497
|
+
- route: recordings/{{recording_id}}/detections
|
|
498
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
Any route can reference any schema directly as `[[schema.table]]`, so a different database (e.g. `owl/5.0`) can query `[[birdnet_2p4.detections]]`. To keep a table private to one database/version, define it in that version's `tables` instead. Qualified references are exposed to DuckDB as real views, so you can also use the full name or your own alias in plain SQL:
|
|
502
|
+
|
|
503
|
+
```yaml
|
|
504
|
+
- route: detections/
|
|
505
|
+
sql: >
|
|
506
|
+
SELECT detections.common_name, COUNT(revisions.id) AS revcount
|
|
507
|
+
FROM [[birdnet_2p4.detections]]
|
|
508
|
+
LEFT JOIN [[revisions]] ON revisions.observation_id = birdnet_2p4.detections.id
|
|
509
|
+
GROUP BY birdnet_2p4.detections.common_name
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
expands to
|
|
513
|
+
|
|
514
|
+
```sql
|
|
515
|
+
SELECT detections.common_name, COUNT(revisions.id) AS revcount
|
|
516
|
+
FROM birdnet_2p4.detections
|
|
517
|
+
LEFT JOIN 's3://your-bucket/birdnet/2.4/revisions.parquet' AS revisions ON revisions.observation_id = birdnet_2p4.detections.id
|
|
518
|
+
GROUP BY birdnet_2p4.detections.common_name
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
Notes:
|
|
522
|
+
- After `FROM`/`JOIN`, `[[schema.table]]` becomes `schema.table` with no alias, so `FROM [[birdnet_2p4.detections]] o` works. Elsewhere it becomes the bare table name (`detections`), because DuckDB doesn't accept `schema.table.*`.
|
|
523
|
+
- Schema and table names used as `[[schema.table]]` must be plain identifiers (letters, digits, underscores).
|
|
524
|
+
- Storage credentials are set per table. Tables whose `region`/`public` differ from the rest get their own S3 secret scoped to their path, so one query can mix regions and public/private buckets.
|
|
525
|
+
- Views are created only for the `[[schema.table]]` tables a query actually references.
|
|
526
|
+
|
|
527
|
+
#### Inline database configs (`slugs`)
|
|
528
|
+
|
|
529
|
+
Simple database/version configs (often just a description and a `schema`) can live in the shared file instead of in their own files. Config files keep working, and the two can be mixed, even for the same database:
|
|
530
|
+
|
|
531
|
+
```yaml
|
|
532
|
+
# api_dock_config/databases/config.yaml
|
|
533
|
+
slugs:
|
|
534
|
+
- name: birdnet-bullfrog # the database slug in the URL
|
|
535
|
+
version: "2.5" # one version...
|
|
536
|
+
description: American Bullfrog Classifier from Birdnet 2.4
|
|
537
|
+
schema: birdnet_bullfrog_2p5v0p5
|
|
538
|
+
- name: birdnet-apple
|
|
539
|
+
authors: [API Team] # ...or several; keys here are defaults for each version
|
|
540
|
+
versions:
|
|
541
|
+
- version: "1.0"
|
|
542
|
+
description: Apple Classifier 1.0
|
|
543
|
+
schema: birdnet_apple_1p0
|
|
544
|
+
- version: "12.0"
|
|
545
|
+
description: Apple Classifier 12.0
|
|
546
|
+
schema: birdnet_apple_12p0
|
|
547
|
+
- name: notes # no version/versions = an unversioned database
|
|
548
|
+
tables:
|
|
549
|
+
notes: s3://your-bucket/notes.parquet
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
- Each entry (or each `versions` item) takes the same keys as a database config file: `description`, `authors`, `schema`, `tables`, `routes`, `query_params`, and so on. Shared `routes`/`query_params` (below), including `include`/`exclude`, apply to them like any other database/version.
|
|
553
|
+
- Like file-based databases, a slug is only served if it's listed under `databases:` in the main `config.yaml`.
|
|
554
|
+
- A database's versions are the union of its version files and its `slugs` versions, so `latest`, the `/{database}` versions listing, and the `expose` catalog endpoints all see both. If a file and a slug define the same database/version, the file wins.
|
|
555
|
+
- Quote versions (`version: "2.10"`). Unquoted YAML numbers are floats, so `2.10` would become `"2.1"`.
|
|
556
|
+
- A malformed `slugs` section (missing `name`, both `version` and `versions`, a duplicate version, or a mix of versioned and unversioned entries for one name) returns a 500 "Shared database configuration error".
|
|
557
|
+
|
|
558
|
+
#### Shared routes and query params
|
|
559
|
+
|
|
560
|
+
The shared file can also define top-level `routes` and `query_params`. These are added to **every** database/version, which is handy when each model/version serves the same endpoints over its own `schema`:
|
|
561
|
+
|
|
562
|
+
```yaml
|
|
563
|
+
# api_dock_config/databases/config.yaml
|
|
564
|
+
database:
|
|
565
|
+
...
|
|
566
|
+
|
|
567
|
+
routes:
|
|
568
|
+
- route: recordings/{{recording_id}}/detections/
|
|
569
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
|
|
570
|
+
- route: detections/{{id}}
|
|
571
|
+
sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].id = {{id}}
|
|
572
|
+
- route: not_for_everyone/{{id}}
|
|
573
|
+
sql: SELECT [[other]].* FROM [[other]] WHERE [[other]].id = {{id}}
|
|
574
|
+
exclude: # don't add this route to these slug/versions
|
|
575
|
+
- 'slug1/3.0'
|
|
576
|
+
- slug: slug2
|
|
577
|
+
version: 2.3
|
|
578
|
+
- slug: slug3
|
|
579
|
+
version: '*' # '*' = every version
|
|
580
|
+
|
|
581
|
+
# the same route defined twice: one for everything except birdnet/2.4, one only for it
|
|
582
|
+
- route: detections/
|
|
583
|
+
exclude: ['birdnet/2.4']
|
|
584
|
+
sql: SELECT [[detections]].* FROM [[detections]]
|
|
585
|
+
- route: detections/
|
|
586
|
+
include: ['birdnet/2.4'] # ONLY add this route to these slug/versions
|
|
587
|
+
sql: SELECT [[detections]].*, [[revisions]].id AS revision_id FROM [[detections]] LEFT JOIN [[revisions]] ON [[revisions]].observation_id = [[detections]].id
|
|
588
|
+
|
|
589
|
+
query_params:
|
|
590
|
+
- confidence:
|
|
591
|
+
sql: "[[detections]].confidence >= {{confidence}}"
|
|
592
|
+
- start_time:
|
|
593
|
+
sql: "[[detections]].start_time >= {{start_time}}"
|
|
594
|
+
exclude: ['slug1/3.9']
|
|
595
|
+
- limit:
|
|
596
|
+
sql_append: LIMIT {{limit}}
|
|
597
|
+
|
|
598
|
+
# limit ALL shared routes / query params to these slug/versions
|
|
599
|
+
route_inclusions: ['birdnet', 'owl/5.0']
|
|
600
|
+
query_inclusions: [] # empty or missing = no restriction
|
|
601
|
+
|
|
602
|
+
# opt slug/versions out of ALL shared routes / query params
|
|
603
|
+
route_exclusions: ['legacy_db']
|
|
604
|
+
query_exclusions:
|
|
605
|
+
- slug: slug4
|
|
606
|
+
version: 1.0
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
Rules:
|
|
610
|
+
- **The version config wins.** Its own routes come first and replace any shared route with the same shape. Shape means the same path segments; `{{param}}` names and leading/trailing slashes are ignored, so `detections/{{id}}` and `/detections/{{detection_id}}/` are the same route. Routes the version config adds on top are kept.
|
|
611
|
+
- Shared `query_params` behave like a version config's top-level `query_params`. They apply to every route, and a param with the same name in the version config (or on a route) overrides the shared one.
|
|
612
|
+
- `include` and `route_inclusions`/`query_inclusions` are the opposite of `exclude` and `route_exclusions`/`query_exclusions`. When given (non-empty), the route or query param is added **only** to the listed slug/versions. A shared item is added only if it passes both the top-level lists and its own `include`/`exclude`.
|
|
613
|
+
- The same route (by shape) or query param (by name) can appear more than once in the shared file. Each database/version gets the first one whose `include`/`exclude` select it, so complementary `include`/`exclude` lists give different databases different versions of an endpoint.
|
|
614
|
+
- `include`, `exclude` and the four top-level lists take a list of `'<slug>/<version>'` strings or `{slug: <slug>, version: <version>}` mappings. `'<slug>'`, `'<slug>/*'` or a missing/`'*'` version match every version, including unversioned databases. Versions compare numerically when possible (`2.3`, `"2.3"`), and `latest` is resolved before matching.
|
|
615
|
+
|
|
397
616
|
**For more details**, see the [SQL Database Support Wiki](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support).
|
|
398
617
|
|
|
399
618
|
---
|