api-dock 0.9.0__tar.gz → 0.9.1__tar.gz

api_dock-0.9.1/PKG-INFO

@@ -0,0 +1,333 @@
+Metadata-Version: 2.4
+Name: api_dock
+Version: 0.9.1
+Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
+Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
+License-Expression: BSD-3-Clause
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: click
+Requires-Dist: fastapi<0.118,>=0.117.1
+Requires-Dist: uvicorn<0.38,>=0.37.0
+Requires-Dist: pyyaml<7,>=6.0.3
+Requires-Dist: httpx<0.29,>=0.28.1
+Requires-Dist: duckdb<2,>=1.1.3
+Requires-Dist: flask
+Requires-Dist: cryptography<49.0.0,>=48.0.0
+Requires-Dist: boto3<2,>=1.42.59
+Provides-Extra: dev
+Requires-Dist: pycodestyle<3,>=2.14.0; extra == "dev"
+Requires-Dist: ipykernel<7,>=6.30.1; extra == "dev"
+Requires-Dist: jupyterlab<5,>=4.4.9; extra == "dev"
+Requires-Dist: numpy<3,>=2.3.3; extra == "dev"
+Requires-Dist: pandas<3,>=2.3.2; extra == "dev"
+Requires-Dist: pytest<10,>=9.0.2; extra == "dev"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]<4,>=3.3.6; extra == "postgres"
+Requires-Dist: psycopg_pool<4,>=3.3.2; extra == "postgres"
+Dynamic: license-file
+
+# API Dock
+
+API Dock builds an API from YAML files. Behind one endpoint it proxies requests to **remote HTTP
+APIs** and serves **SQL routes** over Parquet/CSV files (local disk, S3, GCS, Azure or HTTPS) and
+**PostgreSQL** tables. Run it with the `api-dock` CLI as a FastAPI or Flask app, or embed its
+`RouteMapper` in an existing Python service.
+
+| source | served at | how |
+|---|---|---|
+| remote HTTP API | `/<remote>/<path>` | forwarded upstream and streamed back, with optional allow-lists and restrictions |
+| file tables (Parquet, CSV, ...) | `/<database>/<version>/<route>` | SQL run by DuckDB |
+| PostgreSQL tables | `/<database>/<version>/<route>` | SQL run natively, or by DuckDB when a route mixes sources |
+
+Request values reach the database as bound parameters, never pasted into SQL, and every database
+config is checked when the server starts.
+
+**Documentation lives in the [wiki](https://github.com/SchmidtDSE/api_dock/wiki/).** Start with [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) and
+[Concepts](https://github.com/SchmidtDSE/api_dock/wiki/Concepts).
+
+---
+
+## Install
+
+```bash
+pip install api-dock                     # core
+pip install 'api-dock[postgres]'         # adds the PostgreSQL driver and pool
+conda install -c conda-forge api_dock    # or from conda-forge
+```
+
+API Dock needs Python 3.11 or later.
+
+---
+
+## Quick start
+
+```bash
+api-dock init        # creates api_dock_config/ with commented example files
+```
+
+Replace the examples with a main config, one remote and one database:
+
+```
+api_dock_config/
+├── config.yaml
+├── remotes/
+│   └── httpbin.yaml
+└── databases/
+    └── places.yaml
+```
+
+```yaml
+# api_dock_config/config.yaml
+name: my-api
+description: My first API Dock
+remotes:
+  - httpbin
+databases:
+  - places
+settings:
+  add_trailing_slash: false   # httpbin doesn't want /get/
+```
+
+```yaml
+# api_dock_config/remotes/httpbin.yaml
+name: httpbin
+url: https://httpbin.org
+```
+
+```yaml
+# api_dock_config/databases/places.yaml
+tables:
+  places: data/places.parquet            # or s3://bucket/places/**/*.parquet, a PostgreSQL table, ...
+routes:
+  - route: places
+    sql: SELECT * FROM [[places]]
+    query_params:
+      - country:
+          sql: "country = {{country}}"
+  - route: places/{{id}}
+    sql: SELECT * FROM [[places]] WHERE id = {{id}}
+```
+
+`[[places]]` is replaced by the table's source; `{{id}}` and `{{country}}` are request values,
+sent to the database as bound parameters.
+
+```bash
+api-dock start                                    # FastAPI on port 8000
+curl http://localhost:8000/httpbin/get            # proxied to https://httpbin.org/get
+curl http://localhost:8000/places/places?country=FR
+# [{"id": 2, "name": "Lyon", "country": "FR"}]
+curl http://localhost:8000/places/places/1
+```
+
+[Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) walks through this example, including making the Parquet
+file.
+
+---
+
+## What you can configure
+
+| topic | wiki page |
+|---|---|
+| main config, settings (`timeout`, `base_path`, `duckdb`, ...), multiple configs | [Configuration](https://github.com/SchmidtDSE/api_dock/wiki/Configuration) |
+| versioned remotes and databases, inline versions, `latest` | [Versioning](https://github.com/SchmidtDSE/api_dock/wiki/Versioning) |
+| remote configs (files or `remotes/config.yaml`), allow-lists, `restricted` patterns, route mapping | [Routing and Restrictions](https://github.com/SchmidtDSE/api_dock/wiki/Routing-and-Restrictions) |
+| cookies, database authentication, encrypted values | [Authentication and Cookies](https://github.com/SchmidtDSE/api_dock/wiki/Authentication-and-Cookies) |
+| tables, `[[table]]` references, routes, startup checks | [SQL Database Support](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support) |
+| filtering, sorting, pagination, required params | [Query Parameters](https://github.com/SchmidtDSE/api_dock/wiki/Query-Parameters) |
+| picking SQL from the request | [Conditional SQL](https://github.com/SchmidtDSE/api_dock/wiki/Conditional-SQL) |
+| shared tables, schemas, slugs, shared routes | [Shared Database Config](https://github.com/SchmidtDSE/api_dock/wiki/Shared-Database-Config) |
+| unions across schemas (`[[*.table]]`, schema groups) | [Cross-Schema Queries](https://github.com/SchmidtDSE/api_dock/wiki/Cross-Schema-Queries) |
+| PostgreSQL connections, engines, safety | [PostgreSQL](https://github.com/SchmidtDSE/api_dock/wiki/PostgreSQL) |
+| databases, versions and values generated from a query, refreshed on a schedule | [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups) |
+| `/databases`, `/remotes`, `/sources` listings | [Catalog Endpoints](https://github.com/SchmidtDSE/api_dock/wiki/Catalog-Endpoints) |
+| `RouteMapper` in your own app, production deployment | [Python API and Deployment](https://github.com/SchmidtDSE/api_dock/wiki/Python-API-and-Deployment) |
+
+---
+
+## Example: versions from a catalog (lookups)
+
+When the list of databases/versions lives somewhere else (a table of model runs, a
+deployments API), a lookup turns its rows into config. api_dock runs it at startup, every
+`refresh` and on demand:
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  connections:
+    core: {host: db.example.com, dbname: catalog, user: readonly, password: env:DB_PASSWORD}
+  runs_catalog: {connection: core, table: public.model_runs}   # or {uri: s3://.../runs.parquet}
+
+lookups:
+  model_runs:
+    sql: |
+      SELECT name, version, detections_uri,
+             replace(name, '-', '_') || '_' || replace(version, '.', 'p') AS schema
+      FROM [[runs_catalog]] WHERE published
+    refresh: 7d
+    allow: ["s3://my-bucket/runs/"]        # URIs from rows must start with this
+
+slugs:
+  - from: model_runs                       # one database/version per row
+    name: "{{row.name}}"
+    version: "{{row.version}}"
+    schema:
+      name: "{{row.schema}}"
+      tables:
+        detections: {uri: "{{row.detections_uri}}"}
+```
+
+```yaml
+# api_dock_config/config.yaml
+databases:
+  - from: model_runs                       # serve every database the lookup generates
+settings:
+  lookups:                                 # optional: GET status / POST refresh
+    refresh_route: /admin/lookups
+    token: env:API_DOCK_ADMIN_TOKEN
+```
+
+Lookups can read PostgreSQL tables, Parquet/CSV files or an HTTP API, and can also
+generate remote versions or fill single values. See [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups).
+
+---
+
+## CLI
+
+```bash
+api-dock                                    # list configs and commands
+api-dock init [--force]                     # create api_dock_config/
+api-dock start [config_name]                # serve api_dock_config/<config_name>.yaml (default: config)
+api-dock start --backbone flask --host 127.0.0.1 --port 9000 --log-level debug
+api-dock describe [config_name]             # print the config
+api-dock generate-key                       # local encryption key
+api-dock encrypt "secret"                   # also --method env_key|aws_kms
+api-dock decrypt "gAAAAA..."
+api-dock lookups                            # run the config's lookups and print their rows
+```
+
+Flask responses are buffered and Flask refuses configs with PostgreSQL connections; use the default
+FastAPI backbone for those. Full reference: [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started#cli-reference).
+
+---
+
+## How it works (in brief)
+
+- **Configs.** The main `config.yaml` lists remotes and databases and is read at startup. Remote
+  files, database files and the shared `remotes/config.yaml` and `databases/config.yaml` are read
+  again on each request, so
+  route edits don't need a restart.
+- **Remotes.** A request is checked against the remote's allow/block lists, then forwarded with
+  httpx. The FastAPI app streams the upstream response back.
+- **Databases.** The version is resolved (`latest`, version files, `slugs`), the route matched and
+  its SQL built: `[[table]]` references expanded, query-param fragments appended, values bound.
+- **Engines.** A route whose tables are all on one PostgreSQL connection runs natively through
+  that connection's pool. Anything else runs on an in-memory DuckDB in a worker thread, with
+  PostgreSQL attached read-only when needed.
+- **Lookups.** Named queries (SQL over the configured tables, or an HTTP API) run at startup
+  and on a schedule; `from:` entries turn their rows into database versions or remote versions.
+- **Startup checks.** Every database and version is checked as requests will see it (table
+  references, quoted variables, unions, connections, engines); a bad config stops startup with a
+  message naming the database, version and route.
+
+---
+
+## Repo layout
+
+```
+api_dock/                   the package
+  cli.py                    api-dock commands
+  config.py, config_discovery.py   main/remote config loading, settings, route rules
+  route_mapper.py           RouteMapper: request handling, startup checks, PostgreSQL lifecycle
+  fast_api.py, flask_api.py the two app backbones
+  database_config.py        database configs, shared config, versions and slugs
+  lookups.py                lookups: templates, SQL/HTTP runners, refresh
+  sql_builder.py            SQL building, table references, unions, engine choice
+  database_backends.py      DuckDB backend    postgres_backend.py, postgres_config.py   PostgreSQL
+  storage_auth.py, auth.py, encryption.py, listings.py, sql_template_check.py, types.py
+  example_api_dock_config/  copied by `api-dock init`
+tests/                      pixi run -e dev pytest -q
+```
+
+More in the [Developer Guide](https://github.com/SchmidtDSE/api_dock/wiki/Developer-Guide).
+
+---
+
+# Development
+
+```bash
+pixi install -e dev           # includes psycopg and a local PostgreSQL for the tests
+pixi run -e dev pytest -q     # PostgreSQL tests skip if PostgreSQL/psycopg are missing
+```
+
+## Publishing a Release
+
+Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publish_to_pypi.yml` runs on `release: published`, builds the sdist and wheel with `uv build`, and uploads them using PyPI trusted publishing (OIDC). There's no local build, no API token, and no `twine`. conda-forge follows automatically: its bot opens a version PR on [conda-forge/api_dock-feedstock](https://github.com/conda-forge/api_dock-feedstock), which a maintainer merges.
+
+```bash
+# 0. Start from a clean, up-to-date main
+export VERSION=0.9.1          # the NEW version, no leading "v"
+git checkout main
+git pull origin main
+git status
+
+# 1. Set `version` in pyproject.toml to $VERSION
+
+# 2. Run the tests
+pixi run -e dev pytest -q
+
+# 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
+export COMMIT_MESSAGE='PostgreSQL, lookups, remotes/config.yaml, numeric latest'
+git add -A
+git commit -m "v$VERSION: $COMMIT_MESSAGE"
+git tag "v$VERSION"
+git push origin main "v$VERSION"
+
+# 4. Publish the GitHub Release; this triggers the PyPI upload.
+#    Don't use --draft (the workflow only runs on a published release); no wheel needs attaching.
+gh release create "v$VERSION" \
+  --title "v$VERSION" \
+  --notes "$(cat <<'EOF'
+* new features
+    - PostgreSQL tables (`pip install 'api_dock[postgres]'`): named `connections` in `databases/config.yaml` and `{connection, table}` tables. A route runs natively on PostgreSQL when all its tables are on one connection (one pool per connection, read-only transactions, statement timeout), otherwise on DuckDB with PostgreSQL attached read-only, so unions and joins can mix PostgreSQL and Parquet. Route `engine: postgres|duckdb` checks or forces the choice; 503 when a database is unavailable
+    - Lookups: named queries (SQL over any configured table, or an HTTP API) run at startup, every `refresh` and on demand. `from:` entries turn rows into database/versions (with inline schemas) or remote versions, `{{lookup.<name>.<column>}}` fills single values, and `databases: [- from: <lookup>]` serves what they generate. New rows are checked before use; values used as URIs must match `allow:` prefixes. Optional token-protected refresh endpoint (`settings.lookups`) and `api-dock lookups` CLI
+    - `remotes/config.yaml`: define remotes inline (`version`, `versions` or unversioned); mixes with remote files, files win
+    - Slugs can define their schema inline: `schema: {name, tables}`
+    - Database values convert to JSON recursively, including UUIDs, network addresses, intervals and arrays
+* bug fixes
+    - `latest` and version lists compare versions numerically (`0.10` > `0.9`, `0.10.0` > `0.9.0`) for remotes and databases; they used to compare as floats or text
+* cleanup / other improvements
+    - Database queries go through a `DatabaseBackend` interface (`DuckDBBackend`, `PostgresBackend`); the SQL builder takes the backend's bound-value marker
+    - Startup checks also cover PostgreSQL connections, every route's engine, `remotes/config.yaml` and lookup-generated config
+    - Slim README; detailed docs moved to the wiki (https://github.com/SchmidtDSE/api_dock/wiki), including new Concepts, PostgreSQL, Lookups and Developer Guide pages
+    - Test suite grew from 373 to 626 tests (a throwaway PostgreSQL server runs the PostgreSQL tests when available)
+EOF
+)"
+
+# 5. Watch the publish workflow, then confirm PyPI has the new version
+gh run watch "$(gh run list --workflow=publish_to_pypi.yml -L1 --json databaseId -q '.[0].databaseId')" --repo SchmidtDSE/api_dock
+curl -s https://pypi.org/pypi/api-dock/json | python3 -c "import sys,json; print('PyPI latest:', json.load(sys.stdin)['info']['version'])"
+
+# 6. conda-forge: once the bot opens the v$VERSION PR (usually within hours), check that the recipe's
+#    run requirements match pyproject.toml dependencies (the bot only bumps version + sha256), then merge it
+gh pr list --repo conda-forge/api_dock-feedstock --state open
+```
+
+---
+
+# License
+
+BSD 3-Clause

api_dock-0.9.1/README.md

@@ -0,0 +1,291 @@
+# API Dock
+
+API Dock builds an API from YAML files. Behind one endpoint it proxies requests to **remote HTTP
+APIs** and serves **SQL routes** over Parquet/CSV files (local disk, S3, GCS, Azure or HTTPS) and
+**PostgreSQL** tables. Run it with the `api-dock` CLI as a FastAPI or Flask app, or embed its
+`RouteMapper` in an existing Python service.
+
+| source | served at | how |
+|---|---|---|
+| remote HTTP API | `/<remote>/<path>` | forwarded upstream and streamed back, with optional allow-lists and restrictions |
+| file tables (Parquet, CSV, ...) | `/<database>/<version>/<route>` | SQL run by DuckDB |
+| PostgreSQL tables | `/<database>/<version>/<route>` | SQL run natively, or by DuckDB when a route mixes sources |
+
+Request values reach the database as bound parameters, never pasted into SQL, and every database
+config is checked when the server starts.
+
+**Documentation lives in the [wiki](https://github.com/SchmidtDSE/api_dock/wiki/).** Start with [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) and
+[Concepts](https://github.com/SchmidtDSE/api_dock/wiki/Concepts).
+
+---
+
+## Install
+
+```bash
+pip install api-dock                     # core
+pip install 'api-dock[postgres]'         # adds the PostgreSQL driver and pool
+conda install -c conda-forge api_dock    # or from conda-forge
+```
+
+API Dock needs Python 3.11 or later.
+
+---
+
+## Quick start
+
+```bash
+api-dock init        # creates api_dock_config/ with commented example files
+```
+
+Replace the examples with a main config, one remote and one database:
+
+```
+api_dock_config/
+├── config.yaml
+├── remotes/
+│   └── httpbin.yaml
+└── databases/
+    └── places.yaml
+```
+
+```yaml
+# api_dock_config/config.yaml
+name: my-api
+description: My first API Dock
+remotes:
+  - httpbin
+databases:
+  - places
+settings:
+  add_trailing_slash: false   # httpbin doesn't want /get/
+```
+
+```yaml
+# api_dock_config/remotes/httpbin.yaml
+name: httpbin
+url: https://httpbin.org
+```
+
+```yaml
+# api_dock_config/databases/places.yaml
+tables:
+  places: data/places.parquet            # or s3://bucket/places/**/*.parquet, a PostgreSQL table, ...
+routes:
+  - route: places
+    sql: SELECT * FROM [[places]]
+    query_params:
+      - country:
+          sql: "country = {{country}}"
+  - route: places/{{id}}
+    sql: SELECT * FROM [[places]] WHERE id = {{id}}
+```
+
+`[[places]]` is replaced by the table's source; `{{id}}` and `{{country}}` are request values,
+sent to the database as bound parameters.
+
+```bash
+api-dock start                                    # FastAPI on port 8000
+curl http://localhost:8000/httpbin/get            # proxied to https://httpbin.org/get
+curl http://localhost:8000/places/places?country=FR
+# [{"id": 2, "name": "Lyon", "country": "FR"}]
+curl http://localhost:8000/places/places/1
+```
+
+[Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) walks through this example, including making the Parquet
+file.
+
+---
+
+## What you can configure
+
+| topic | wiki page |
+|---|---|
+| main config, settings (`timeout`, `base_path`, `duckdb`, ...), multiple configs | [Configuration](https://github.com/SchmidtDSE/api_dock/wiki/Configuration) |
+| versioned remotes and databases, inline versions, `latest` | [Versioning](https://github.com/SchmidtDSE/api_dock/wiki/Versioning) |
+| remote configs (files or `remotes/config.yaml`), allow-lists, `restricted` patterns, route mapping | [Routing and Restrictions](https://github.com/SchmidtDSE/api_dock/wiki/Routing-and-Restrictions) |
+| cookies, database authentication, encrypted values | [Authentication and Cookies](https://github.com/SchmidtDSE/api_dock/wiki/Authentication-and-Cookies) |
+| tables, `[[table]]` references, routes, startup checks | [SQL Database Support](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support) |
+| filtering, sorting, pagination, required params | [Query Parameters](https://github.com/SchmidtDSE/api_dock/wiki/Query-Parameters) |
+| picking SQL from the request | [Conditional SQL](https://github.com/SchmidtDSE/api_dock/wiki/Conditional-SQL) |
+| shared tables, schemas, slugs, shared routes | [Shared Database Config](https://github.com/SchmidtDSE/api_dock/wiki/Shared-Database-Config) |
+| unions across schemas (`[[*.table]]`, schema groups) | [Cross-Schema Queries](https://github.com/SchmidtDSE/api_dock/wiki/Cross-Schema-Queries) |
+| PostgreSQL connections, engines, safety | [PostgreSQL](https://github.com/SchmidtDSE/api_dock/wiki/PostgreSQL) |
+| databases, versions and values generated from a query, refreshed on a schedule | [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups) |
+| `/databases`, `/remotes`, `/sources` listings | [Catalog Endpoints](https://github.com/SchmidtDSE/api_dock/wiki/Catalog-Endpoints) |
+| `RouteMapper` in your own app, production deployment | [Python API and Deployment](https://github.com/SchmidtDSE/api_dock/wiki/Python-API-and-Deployment) |
+
+---
+
+## Example: versions from a catalog (lookups)
+
+When the list of databases/versions lives somewhere else (a table of model runs, a
+deployments API), a lookup turns its rows into config. api_dock runs it at startup, every
+`refresh` and on demand:
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  connections:
+    core: {host: db.example.com, dbname: catalog, user: readonly, password: env:DB_PASSWORD}
+  runs_catalog: {connection: core, table: public.model_runs}   # or {uri: s3://.../runs.parquet}
+
+lookups:
+  model_runs:
+    sql: |
+      SELECT name, version, detections_uri,
+             replace(name, '-', '_') || '_' || replace(version, '.', 'p') AS schema
+      FROM [[runs_catalog]] WHERE published
+    refresh: 7d
+    allow: ["s3://my-bucket/runs/"]        # URIs from rows must start with this
+
+slugs:
+  - from: model_runs                       # one database/version per row
+    name: "{{row.name}}"
+    version: "{{row.version}}"
+    schema:
+      name: "{{row.schema}}"
+      tables:
+        detections: {uri: "{{row.detections_uri}}"}
+```
+
+```yaml
+# api_dock_config/config.yaml
+databases:
+  - from: model_runs                       # serve every database the lookup generates
+settings:
+  lookups:                                 # optional: GET status / POST refresh
+    refresh_route: /admin/lookups
+    token: env:API_DOCK_ADMIN_TOKEN
+```
+
+Lookups can read PostgreSQL tables, Parquet/CSV files or an HTTP API, and can also
+generate remote versions or fill single values. See [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups).
+
+---
+
+## CLI
+
+```bash
+api-dock                                    # list configs and commands
+api-dock init [--force]                     # create api_dock_config/
+api-dock start [config_name]                # serve api_dock_config/<config_name>.yaml (default: config)
+api-dock start --backbone flask --host 127.0.0.1 --port 9000 --log-level debug
+api-dock describe [config_name]             # print the config
+api-dock generate-key                       # local encryption key
+api-dock encrypt "secret"                   # also --method env_key|aws_kms
+api-dock decrypt "gAAAAA..."
+api-dock lookups                            # run the config's lookups and print their rows
+```
+
+Flask responses are buffered and Flask refuses configs with PostgreSQL connections; use the default
+FastAPI backbone for those. Full reference: [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started#cli-reference).
+
+---
+
+## How it works (in brief)
+
+- **Configs.** The main `config.yaml` lists remotes and databases and is read at startup. Remote
+  files, database files and the shared `remotes/config.yaml` and `databases/config.yaml` are read
+  again on each request, so
+  route edits don't need a restart.
+- **Remotes.** A request is checked against the remote's allow/block lists, then forwarded with
+  httpx. The FastAPI app streams the upstream response back.
+- **Databases.** The version is resolved (`latest`, version files, `slugs`), the route matched and
+  its SQL built: `[[table]]` references expanded, query-param fragments appended, values bound.
+- **Engines.** A route whose tables are all on one PostgreSQL connection runs natively through
+  that connection's pool. Anything else runs on an in-memory DuckDB in a worker thread, with
+  PostgreSQL attached read-only when needed.
+- **Lookups.** Named queries (SQL over the configured tables, or an HTTP API) run at startup
+  and on a schedule; `from:` entries turn their rows into database versions or remote versions.
+- **Startup checks.** Every database and version is checked as requests will see it (table
+  references, quoted variables, unions, connections, engines); a bad config stops startup with a
+  message naming the database, version and route.
+
+---
+
+## Repo layout
+
+```
+api_dock/                   the package
+  cli.py                    api-dock commands
+  config.py, config_discovery.py   main/remote config loading, settings, route rules
+  route_mapper.py           RouteMapper: request handling, startup checks, PostgreSQL lifecycle
+  fast_api.py, flask_api.py the two app backbones
+  database_config.py        database configs, shared config, versions and slugs
+  lookups.py                lookups: templates, SQL/HTTP runners, refresh
+  sql_builder.py            SQL building, table references, unions, engine choice
+  database_backends.py      DuckDB backend    postgres_backend.py, postgres_config.py   PostgreSQL
+  storage_auth.py, auth.py, encryption.py, listings.py, sql_template_check.py, types.py
+  example_api_dock_config/  copied by `api-dock init`
+tests/                      pixi run -e dev pytest -q
+```
+
+More in the [Developer Guide](https://github.com/SchmidtDSE/api_dock/wiki/Developer-Guide).
+
+---
+
+# Development
+
+```bash
+pixi install -e dev           # includes psycopg and a local PostgreSQL for the tests
+pixi run -e dev pytest -q     # PostgreSQL tests skip if PostgreSQL/psycopg are missing
+```
+
+## Publishing a Release
+
+Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publish_to_pypi.yml` runs on `release: published`, builds the sdist and wheel with `uv build`, and uploads them using PyPI trusted publishing (OIDC). There's no local build, no API token, and no `twine`. conda-forge follows automatically: its bot opens a version PR on [conda-forge/api_dock-feedstock](https://github.com/conda-forge/api_dock-feedstock), which a maintainer merges.
+
+```bash
+# 0. Start from a clean, up-to-date main
+export VERSION=0.9.1          # the NEW version, no leading "v"
+git checkout main
+git pull origin main
+git status
+
+# 1. Set `version` in pyproject.toml to $VERSION
+
+# 2. Run the tests
+pixi run -e dev pytest -q
+
+# 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
+export COMMIT_MESSAGE='PostgreSQL, lookups, remotes/config.yaml, numeric latest'
+git add -A
+git commit -m "v$VERSION: $COMMIT_MESSAGE"
+git tag "v$VERSION"
+git push origin main "v$VERSION"
+
+# 4. Publish the GitHub Release; this triggers the PyPI upload.
+#    Don't use --draft (the workflow only runs on a published release); no wheel needs attaching.
+gh release create "v$VERSION" \
+  --title "v$VERSION" \
+  --notes "$(cat <<'EOF'
+* new features
+    - PostgreSQL tables (`pip install 'api_dock[postgres]'`): named `connections` in `databases/config.yaml` and `{connection, table}` tables. A route runs natively on PostgreSQL when all its tables are on one connection (one pool per connection, read-only transactions, statement timeout), otherwise on DuckDB with PostgreSQL attached read-only, so unions and joins can mix PostgreSQL and Parquet. Route `engine: postgres|duckdb` checks or forces the choice; 503 when a database is unavailable
+    - Lookups: named queries (SQL over any configured table, or an HTTP API) run at startup, every `refresh` and on demand. `from:` entries turn rows into database/versions (with inline schemas) or remote versions, `{{lookup.<name>.<column>}}` fills single values, and `databases: [- from: <lookup>]` serves what they generate. New rows are checked before use; values used as URIs must match `allow:` prefixes. Optional token-protected refresh endpoint (`settings.lookups`) and `api-dock lookups` CLI
+    - `remotes/config.yaml`: define remotes inline (`version`, `versions` or unversioned); mixes with remote files, files win
+    - Slugs can define their schema inline: `schema: {name, tables}`
+    - Database values convert to JSON recursively, including UUIDs, network addresses, intervals and arrays
+* bug fixes
+    - `latest` and version lists compare versions numerically (`0.10` > `0.9`, `0.10.0` > `0.9.0`) for remotes and databases; they used to compare as floats or text
+* cleanup / other improvements
+    - Database queries go through a `DatabaseBackend` interface (`DuckDBBackend`, `PostgresBackend`); the SQL builder takes the backend's bound-value marker
+    - Startup checks also cover PostgreSQL connections, every route's engine, `remotes/config.yaml` and lookup-generated config
+    - Slim README; detailed docs moved to the wiki (https://github.com/SchmidtDSE/api_dock/wiki), including new Concepts, PostgreSQL, Lookups and Developer Guide pages
+    - Test suite grew from 373 to 626 tests (a throwaway PostgreSQL server runs the PostgreSQL tests when available)
+EOF
+)"
+
+# 5. Watch the publish workflow, then confirm PyPI has the new version
+gh run watch "$(gh run list --workflow=publish_to_pypi.yml -L1 --json databaseId -q '.[0].databaseId')" --repo SchmidtDSE/api_dock
+curl -s https://pypi.org/pypi/api-dock/json | python3 -c "import sys,json; print('PyPI latest:', json.load(sys.stdin)['info']['version'])"
+
+# 6. conda-forge: once the bot opens the v$VERSION PR (usually within hours), check that the recipe's
+#    run requirements match pyproject.toml dependencies (the bot only bumps version + sha256), then merge it
+gh pr list --repo conda-forge/api_dock-feedstock --state open
+```
+
+---
+
+# License
+
+BSD 3-Clause