api-dock 0.7.0__tar.gz → 0.8.0__tar.gz

{api_dock-0.7.0 → api_dock-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: api_dock
-Version: 0.7.0
+Version: 0.8.0
 Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
 Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
 License-Expression: BSD-3-Clause
@@ -95,6 +95,7 @@ Here is an example:
 api_dock_config
 ├── config.yaml               # The default main-config file
 ├── databases
+│    ├── config.yaml          # (optional) shared tables/schemas for all databases
 │    ├── unversioned_db.yaml  # Database config without versioning
 │    └── versioned_db         # Folder containing database configs for different versions
 │        ├── 0.1.yaml
@@ -231,6 +232,58 @@ The optional `settings` section controls HTTP behavior:
 
 - **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
 
+### Catalog Endpoints (`expose`)
+
+The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
+
+```yaml
+# Enable all three defaults: /databases, /remotes, /sources
+expose: true
+```
+
+```yaml
+GET /databases
+# [{"model": "birdnet", "version": "2.4"},
+#  {"model": "birdnet", "version": "3.0"},
+#  {"model": "owl", "version": "0.5"}]
+
+GET /sources        # databases + remotes, combined
+# [{"model": "birdnet", "version": "2.4"}, ..., {"model": "core", "version": "0.5.0"}]
+```
+
+Versions are the config filename stems, so semver like `0.5.0` is preserved; unversioned sources report `version: null`. Enable only what you want, and control the output shape with `dict`:
+
+```yaml
+expose:
+  dict: false          # return "model/version" strings instead of {model, version} dicts
+  databases: true      # add /databases
+  remotes: true        # add /remotes
+  # sources omitted → not added
+
+# GET /databases -> ["birdnet/2.4", "birdnet/3.0", "owl/0.5"]
+```
+
+Each of `databases` / `remotes` / `sources` accepts several forms:
+
+```yaml
+expose:
+  databases: false                     # do not add the endpoint
+
+  remotes: "list/remotes"              # custom route path (same as true, but at /list/remotes)
+
+  databases:                           # explicit list of models (optionally version-filtered)
+    - birdnet
+    - owl:
+        versions: [4.0, 5.0]           # ints/floats/strings all match the "4.0"/"5.0" stems
+
+  sources:                             # most explicit form
+    route: "list/sources"
+    include: [birdnet, core]           # true (default) | false | list of models
+    dict: true                         # per-endpoint override of the top-level `dict`
+```
+
+Each listing route is served both with and without a trailing slash (e.g. `/sources` and `/sources/` both work), so it doesn't matter which convention your clients use. Custom multi-segment routes (e.g. `list/databases`) take precedence over the `/{remote}/{path}` proxy. If a listing route would shadow a configured remote/database, or an `include` names something that doesn't exist, API Dock emits a startup warning. The exposed routes are also reflected in the root (`/`) metadata's `endpoints`.
+
 ---
 
 ## Remote Configurations
@@ -356,6 +409,7 @@ Database configurations are stored in `api_dock_config/databases/` directory. Ea
 - **tables**: Mapping of table names to file paths (supports S3, GCS, HTTPS, local paths)
 - **queries**: Named SQL queries for reuse
 - **routes**: REST endpoints mapped to SQL queries
+- **schema** (optional): the shared schema (from `databases/config.yaml`) this config's `[[table]]` references fall back to. See [Shared Tables and Schemas](#shared-tables-and-schemas-databasesconfigyaml)
 
 ### Syntax
 
@@ -433,6 +487,171 @@ routes:
     sql: "[[get_permissions]]"
 ```
 
+### Shared Tables and Schemas (`databases/config.yaml`)
+
+When several databases or versions read the same tables, define them once in the optional `api_dock_config/databases/config.yaml`. Everything lives under a `database` key: `meta` holds default table metadata, `schema` holds named groups of tables, and every other key is a global table.
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  # global tables, available as [[table1]] in any database config
+  table1:
+    uri: s3://your-bucket/table1.parquet
+  table3:
+    uri: s3://your-other-bucket/table3.parquet
+    region: us-west-1        # a table's own keys override `meta`
+    public: false
+
+  # defaults applied to every table (shared tables and the tables in each version config)
+  meta:
+    region: us-west-2
+    public: true
+
+  # schemas, available as [[birdnet_2p4.detections]] in any route
+  schema:
+    birdnet_2p4:
+      detections:
+        uri: s3://your-bucket/birdnet/2.4/detections.parquet
+    birdnet_3p0:
+      detections:
+        uri: s3://your-bucket/birdnet/3.0/detections.parquet
+        public: false
+```
+
+A version config can name the schema it uses with `schema:`. An unqualified `[[name]]` is then looked up in order, first match wins:
+
+1. the version config's own `tables`
+2. its `schema:` in the shared config
+3. the shared config's global tables
+
+```yaml
+# api_dock_config/databases/birdnet/2.4.yaml
+name: birdnet
+schema: birdnet_2p4
+tables:
+  revisions: s3://your-bucket/birdnet/2.4/revisions.parquet   # local to this version
+
+routes:
+  # [[detections]] isn't in `tables`, so it comes from the birdnet_2p4 schema
+  - route: recordings/{{recording_id}}/detections
+    sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
+```
+
+Any route can reference any schema directly as `[[schema.table]]`, so a different database (e.g. `owl/5.0`) can query `[[birdnet_2p4.detections]]`. To keep a table private to one database/version, define it in that version's `tables` instead. Qualified references are exposed to DuckDB as real views, so you can also use the full name or your own alias in plain SQL:
+
+```yaml
+  - route: detections/
+    sql: >
+      SELECT detections.common_name, COUNT(revisions.id) AS revcount
+      FROM [[birdnet_2p4.detections]]
+      LEFT JOIN [[revisions]] ON revisions.observation_id = birdnet_2p4.detections.id
+      GROUP BY birdnet_2p4.detections.common_name
+```
+
+expands to
+
+```sql
+SELECT detections.common_name, COUNT(revisions.id) AS revcount
+FROM birdnet_2p4.detections
+LEFT JOIN 's3://your-bucket/birdnet/2.4/revisions.parquet' AS revisions ON revisions.observation_id = birdnet_2p4.detections.id
+GROUP BY birdnet_2p4.detections.common_name
+```
+
+Notes:
+- After `FROM`/`JOIN`, `[[schema.table]]` becomes `schema.table` with no alias, so `FROM [[birdnet_2p4.detections]] o` works. Elsewhere it becomes the bare table name (`detections`), because DuckDB doesn't accept `schema.table.*`.
+- Schema and table names used as `[[schema.table]]` must be plain identifiers (letters, digits, underscores).
+- Storage credentials are set per table. Tables whose `region`/`public` differ from the rest get their own S3 secret scoped to their path, so one query can mix regions and public/private buckets.
+- Views are created only for the `[[schema.table]]` tables a query actually references.
+
+#### Inline database configs (`slugs`)
+
+Simple database/version configs (often just a description and a `schema`) can live in the shared file instead of in their own files. Config files keep working, and the two can be mixed, even for the same database:
+
+```yaml
+# api_dock_config/databases/config.yaml
+slugs:
+  - name: birdnet-bullfrog         # the database slug in the URL
+    version: "2.5"                 # one version...
+    description: American Bullfrog Classifier from Birdnet 2.4
+    schema: birdnet_bullfrog_2p5v0p5
+  - name: birdnet-apple
+    authors: [API Team]            # ...or several; keys here are defaults for each version
+    versions:
+      - version: "1.0"
+        description: Apple Classifier 1.0
+        schema: birdnet_apple_1p0
+      - version: "12.0"
+        description: Apple Classifier 12.0
+        schema: birdnet_apple_12p0
+  - name: notes                    # no version/versions = an unversioned database
+    tables:
+      notes: s3://your-bucket/notes.parquet
+```
+
+- Each entry (or each `versions` item) takes the same keys as a database config file: `description`, `authors`, `schema`, `tables`, `routes`, `query_params`, and so on. Shared `routes`/`query_params` (below), including `include`/`exclude`, apply to them like any other database/version.
+- Like file-based databases, a slug is only served if it's listed under `databases:` in the main `config.yaml`.
+- A database's versions are the union of its version files and its `slugs` versions, so `latest`, the `/{database}` versions listing, and the `expose` catalog endpoints all see both. If a file and a slug define the same database/version, the file wins.
+- Quote versions (`version: "2.10"`). Unquoted YAML numbers are floats, so `2.10` would become `"2.1"`.
+- A malformed `slugs` section (missing `name`, both `version` and `versions`, a duplicate version, or a mix of versioned and unversioned entries for one name) returns a 500 "Shared database configuration error".
+
+#### Shared routes and query params
+
+The shared file can also define top-level `routes` and `query_params`. These are added to **every** database/version, which is handy when each model/version serves the same endpoints over its own `schema`:
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  ...
+
+routes:
+  - route: recordings/{{recording_id}}/detections/
+    sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].recording_id = {{recording_id}}
+  - route: detections/{{id}}
+    sql: SELECT [[detections]].* FROM [[detections]] WHERE [[detections]].id = {{id}}
+  - route: not_for_everyone/{{id}}
+    sql: SELECT [[other]].* FROM [[other]] WHERE [[other]].id = {{id}}
+    exclude:                       # don't add this route to these slug/versions
+      - 'slug1/3.0'
+      - slug: slug2
+        version: 2.3
+      - slug: slug3
+        version: '*'               # '*' = every version
+
+  # the same route defined twice: one for everything except birdnet/2.4, one only for it
+  - route: detections/
+    exclude: ['birdnet/2.4']
+    sql: SELECT [[detections]].* FROM [[detections]]
+  - route: detections/
+    include: ['birdnet/2.4']       # ONLY add this route to these slug/versions
+    sql: SELECT [[detections]].*, [[revisions]].id AS revision_id FROM [[detections]] LEFT JOIN [[revisions]] ON [[revisions]].observation_id = [[detections]].id
+
+query_params:
+  - confidence:
+      sql: "[[detections]].confidence >= {{confidence}}"
+  - start_time:
+      sql: "[[detections]].start_time >= {{start_time}}"
+      exclude: ['slug1/3.9']
+  - limit:
+      sql_append: LIMIT {{limit}}
+
+# limit ALL shared routes / query params to these slug/versions
+route_inclusions: ['birdnet', 'owl/5.0']
+query_inclusions: []                 # empty or missing = no restriction
+
+# opt slug/versions out of ALL shared routes / query params
+route_exclusions: ['legacy_db']
+query_exclusions:
+  - slug: slug4
+    version: 1.0
+```
+
+Rules:
+- **The version config wins.** Its own routes come first and replace any shared route with the same shape. Shape means the same path segments; `{{param}}` names and leading/trailing slashes are ignored, so `detections/{{id}}` and `/detections/{{detection_id}}/` are the same route. Routes the version config adds on top are kept.
+- Shared `query_params` behave like a version config's top-level `query_params`. They apply to every route, and a param with the same name in the version config (or on a route) overrides the shared one.
+- `include` and `route_inclusions`/`query_inclusions` are the opposite of `exclude` and `route_exclusions`/`query_exclusions`. When given (non-empty), the route or query param is added **only** to the listed slug/versions. A shared item is added only if it passes both the top-level lists and its own `include`/`exclude`.
+- The same route (by shape) or query param (by name) can appear more than once in the shared file. Each database/version gets the first one whose `include`/`exclude` select it, so complementary `include`/`exclude` lists give different databases different versions of an endpoint.
+- `include`, `exclude` and the four top-level lists take a list of `'<slug>/<version>'` strings or `{slug: <slug>, version: <version>}` mappings. `'<slug>'`, `'<slug>/*'` or a missing/`'*'` version match every version, including unversioned databases. Versions compare numerically when possible (`2.3`, `"2.3"`), and `latest` is resolved before matching.
+
 **For more details**, see the [SQL Database Support Wiki](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support).
 
 ---
@@ -672,6 +891,125 @@ Parameters are processed in this order (first match wins for early returns):
 
 ---
 
+## Conditional SQL Selection
+
+Some routes need a *different* base query depending on the request — for example, `?count=true` should return a species histogram (`SELECT … COUNT(*) … GROUP BY …`) rather than rows. `sql_append` can't help (it only adds trailing clauses), and a second `route:` can't either (the path is identical). For this, a route's `sql` may be a **rule list** instead of a string: a first-match-wins decision tree that picks the base query from the presence and value of path, query, and cookie params.
+
+Everything downstream is unchanged: the selected base composes with `query_params` WHERE-fragments and `sql_append` exactly as a plain `sql:` string does.
+
+### The histogram example
+
+```yaml
+routes:
+  - route: detections
+    sql:
+      # ?count=<truthy>  → species histogram
+      - when: count
+        then:
+          sql: >
+            SELECT [[detections]].common_name, [[detections]].scientific_name,
+                   COUNT(*) AS count
+            FROM [[detections]]
+          sql_append: GROUP BY [[detections]].common_name, [[detections]].scientific_name
+      # otherwise → detection rows
+      - else: SELECT [[detections]].* FROM [[detections]]
+    query_params:
+      - recording:
+          sql: "[[detections]].recording_id = {{recording}}"
+          multivalue_sql: "[[detections]].recording_id IN {{recording}}"
+      - limit:
+          sql_append: LIMIT {{limit}}          # applies in BOTH modes
+```
+
+```bash
+GET /db/detections?recording=1&count=true
+# SELECT detections.common_name, detections.scientific_name, COUNT(*) AS count
+#   FROM detections WHERE detections.recording_id = '1'
+#   GROUP BY detections.common_name, detections.scientific_name
+
+GET /db/detections?recording=1
+# SELECT detections.* FROM detections WHERE detections.recording_id = '1'
+```
+
+Note the pipeline order: the selected branch's `sql_append` (the `GROUP BY`) is applied **before** route-level `sql_append` (the shared `LIMIT`), so SQL clause order stays valid.
+
+### Rule forms
+
+A `sql` list contains rules evaluated top to bottom; the **first match wins**. Each rule's payload (an *sql node*) is a SQL string, a leaf object `{sql, sql_append}`, or a nested rule list.
+
+```yaml
+sql:
+  - when: count                       # fires when `count` is truthy (shorthand for equals: _truthy)
+    then: <sql node>
+
+  - when: mode
+    equals: 'true'                    # fires only when mode == "true" (case-insensitive)
+    then: <sql node>
+
+  - when: format                      # value map: different SQL per value
+    match:
+      species: <sql node>             # ?format=species
+      recording: <sql node>           # ?format=recording
+      _truthy: <sql node>             # any other truthy value
+      _default: <sql node>            # any present value not matched above
+
+  - when: [count, recording]          # list: fires when ALL are truthy (AND)
+    then: <sql node>
+
+  - when: [count, something_else]     # list + positional case list
+    match:
+      - values: [_any, x]             # something_else == x, count anything
+        then: <sql node>
+      - values: [_truthy, _absent]    # count truthy AND something_else not passed
+        then: <sql node>
+      - default: <sql node>
+
+  - else: <sql node>                  # default (a trailing bare string works too)
+```
+
+Nesting works because a payload can itself be a rule list:
+
+```yaml
+sql:
+  - when: some_value
+    match:
+      '4':
+        - when: count
+          then: <sql for value 4 with count>
+        - else: <sql for value 4>
+      _default: <sql for other values>
+  - else: <base sql>
+```
+
+### Value specs
+
+| spec | matches when the param… |
+|---|---|
+| `'literal'` (`'4'`, `'true'`) | is present and equals it (case-insensitive) |
+| `_truthy` / `_falsy` | present, and value is / isn't in `{"", "0", "false", "no", "off", "null", "none"}` |
+| `_present` / `_absent` | exists / does not exist |
+| `_any` | wildcard — present or absent (used for a position in a case list) |
+| `_default` | catch-all for any *present* value (value maps only) |
+
+In a single-param `match:` map, precedence is order-independent: exact literal > `_falsy`/`_truthy` > `_present`/`_absent` > `_default`. In a positional case list, cases match strictly top-to-bottom.
+
+### No match → URL error
+
+If no rule matches and there is no default (`else`, a trailing bare string, or a `_default`/`default` catch-all), the request returns a **400** with `{"error": "No matching query configuration for the given parameters", "http_status": 400}`. Customize it with a terminal `no_match` rule:
+
+```yaml
+sql:
+  - when: recording
+    then: SELECT [[detections]].* FROM [[detections]] WHERE recording_id = {{recording}}
+  - no_match:
+      error: "recording is required, or pass count=true for a histogram"
+      http_status: 400
+```
+
+Cookies participate via the `cookies.<name>` key (e.g. `when: cookies.role`, `equals: admin`).
+
+---
+
 # CLI
 
 ## Commands