api-dock 0.9.0__tar.gz → 0.10.0__tar.gz

api_dock-0.10.0/PKG-INFO

@@ -0,0 +1,347 @@
+Metadata-Version: 2.4
+Name: api_dock
+Version: 0.10.0
+Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
+Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
+License-Expression: BSD-3-Clause
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: click
+Requires-Dist: fastapi<0.118,>=0.117.1
+Requires-Dist: uvicorn<0.38,>=0.37.0
+Requires-Dist: pyyaml<7,>=6.0.3
+Requires-Dist: httpx<0.29,>=0.28.1
+Requires-Dist: duckdb<2,>=1.1.3
+Requires-Dist: flask
+Requires-Dist: cryptography<49.0.0,>=48.0.0
+Requires-Dist: boto3<2,>=1.42.59
+Provides-Extra: dev
+Requires-Dist: pycodestyle<3,>=2.14.0; extra == "dev"
+Requires-Dist: ipykernel<7,>=6.30.1; extra == "dev"
+Requires-Dist: jupyterlab<5,>=4.4.9; extra == "dev"
+Requires-Dist: numpy<3,>=2.3.3; extra == "dev"
+Requires-Dist: pandas<3,>=2.3.2; extra == "dev"
+Requires-Dist: pytest<10,>=9.0.2; extra == "dev"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]<4,>=3.3.6; extra == "postgres"
+Requires-Dist: psycopg_pool<4,>=3.3.2; extra == "postgres"
+Provides-Extra: gcp
+Requires-Dist: google-cloud-secret-manager<3,>=2; extra == "gcp"
+Dynamic: license-file
+
+# API Dock
+
+API Dock builds an API from YAML files. Behind one endpoint it proxies requests to **remote HTTP
+APIs** and serves **SQL routes** over Parquet/CSV files (local disk, S3, GCS, Azure or HTTPS) and
+**PostgreSQL** tables. Run it with the `api-dock` CLI as a FastAPI or Flask app, or embed its
+`RouteMapper` in an existing Python service.
+
+| source | served at | how |
+|---|---|---|
+| remote HTTP API | `/<remote>/<path>` | forwarded upstream and streamed back, with optional allow-lists and restrictions |
+| file tables (Parquet, CSV, ...) | `/<database>/<version>/<route>` | SQL run by DuckDB |
+| PostgreSQL tables | `/<database>/<version>/<route>` | SQL run natively, or by DuckDB when a route mixes sources |
+
+Request values reach the database as bound parameters, never pasted into SQL, and every database
+config is checked when the server starts.
+
+**Documentation lives in the [wiki](https://github.com/SchmidtDSE/api_dock/wiki/).** Start with [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) and
+[Concepts](https://github.com/SchmidtDSE/api_dock/wiki/Concepts).
+
+---
+
+## Install
+
+```bash
+pip install api-dock                     # core
+pip install 'api-dock[postgres]'         # adds the PostgreSQL driver and pool
+conda install -c conda-forge api_dock    # or from conda-forge
+```
+
+API Dock needs Python 3.11 or later.
+
+---
+
+## Quick start
+
+```bash
+api-dock init        # creates api_dock_config/ with commented example files
+```
+
+Replace the examples with a main config, one remote and one database:
+
+```
+api_dock_config/
+├── config.yaml
+├── remotes/
+│   └── httpbin.yaml
+└── databases/
+    └── places.yaml
+```
+
+```yaml
+# api_dock_config/config.yaml
+name: my-api
+description: My first API Dock
+remotes:
+  - httpbin
+databases:
+  - places
+settings:
+  add_trailing_slash: false   # httpbin doesn't want /get/
+```
+
+```yaml
+# api_dock_config/remotes/httpbin.yaml
+name: httpbin
+url: https://httpbin.org
+```
+
+```yaml
+# api_dock_config/databases/places.yaml
+tables:
+  places: data/places.parquet            # or s3://bucket/places/**/*.parquet, a PostgreSQL table, ...
+routes:
+  - route: places
+    sql: SELECT * FROM [[places]]
+    query_params:
+      - country:
+          sql: "country = {{country}}"
+  - route: places/{{id}}
+    sql: SELECT * FROM [[places]] WHERE id = {{id}}
+```
+
+`[[places]]` is replaced by the table's source; `{{id}}` and `{{country}}` are request values,
+sent to the database as bound parameters.
+
+```bash
+api-dock start                                    # FastAPI on port 8000
+curl http://localhost:8000/httpbin/get            # proxied to https://httpbin.org/get
+curl http://localhost:8000/places/places?country=FR
+# [{"id": 2, "name": "Lyon", "country": "FR"}]
+curl http://localhost:8000/places/places/1
+```
+
+[Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) walks through this example, including making the Parquet
+file.
+
+---
+
+## What you can configure
+
+| topic | wiki page |
+|---|---|
+| main config, settings (`timeout`, `base_path`, `duckdb`, ...), multiple configs | [Configuration](https://github.com/SchmidtDSE/api_dock/wiki/Configuration) |
+| versioned remotes and databases, inline versions, `latest` | [Versioning](https://github.com/SchmidtDSE/api_dock/wiki/Versioning) |
+| remote configs (files or `remotes/config.yaml`), allow-lists, `restricted` patterns, route mapping | [Routing and Restrictions](https://github.com/SchmidtDSE/api_dock/wiki/Routing-and-Restrictions) |
+| cookies, database authentication, encrypted values | [Authentication and Cookies](https://github.com/SchmidtDSE/api_dock/wiki/Authentication-and-Cookies) |
+| tables, `[[table]]` references, routes, startup checks | [SQL Database Support](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support) |
+| filtering, sorting, pagination, required params | [Query Parameters](https://github.com/SchmidtDSE/api_dock/wiki/Query-Parameters) |
+| picking SQL from the request | [Conditional SQL](https://github.com/SchmidtDSE/api_dock/wiki/Conditional-SQL) |
+| shared tables, schemas, slugs, shared routes | [Shared Database Config](https://github.com/SchmidtDSE/api_dock/wiki/Shared-Database-Config) |
+| unions across schemas (`[[*.table]]`, schema groups) | [Cross-Schema Queries](https://github.com/SchmidtDSE/api_dock/wiki/Cross-Schema-Queries) |
+| PostgreSQL connections, engines, safety | [PostgreSQL](https://github.com/SchmidtDSE/api_dock/wiki/PostgreSQL) |
+| databases, versions and values generated from a query, refreshed on a schedule | [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups) |
+| `/databases`, `/remotes`, `/sources` listings | [Catalog Endpoints](https://github.com/SchmidtDSE/api_dock/wiki/Catalog-Endpoints) |
+| `RouteMapper` in your own app, production deployment | [Python API and Deployment](https://github.com/SchmidtDSE/api_dock/wiki/Python-API-and-Deployment) |
+
+---
+
+## Example: versions from a catalog (lookups)
+
+When the list of databases/versions lives somewhere else (a table of model runs, a
+deployments API), a lookup turns its rows into config. api_dock runs it at startup, every
+`refresh` and on demand:
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  connections:
+    core: {host: db.example.com, dbname: catalog, user: readonly, password: env:DB_PASSWORD}
+  runs_catalog: {connection: core, table: public.model_runs}   # or {uri: s3://.../runs.parquet}
+
+lookups:
+  model_runs:
+    sql: |
+      SELECT name, version, detections_uri,
+             replace(name, '-', '_') || '_' || replace(version, '.', 'p') AS schema
+      FROM [[runs_catalog]] WHERE published
+    refresh: 7d
+    allow: ["s3://my-bucket/runs/"]        # URIs from rows must start with this
+
+slugs:
+  - from: model_runs                       # one database/version per row
+    name: "{{row.name}}"
+    version: "{{row.version}}"
+    schema:
+      name: "{{row.schema}}"
+      tables:
+        detections: {uri: "{{row.detections_uri}}"}
+```
+
+```yaml
+# api_dock_config/config.yaml
+databases:
+  - from: model_runs                       # serve every database the lookup generates
+settings:
+  lookups:                                 # optional: GET status / POST refresh
+    refresh_route: /admin/lookups
+    token: env:API_DOCK_ADMIN_TOKEN
+```
+
+Lookups can read PostgreSQL tables, Parquet/CSV files or an HTTP API, and can also
+generate remote versions or fill single values. See [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups).
+
+---
+
+## CLI
+
+```bash
+api-dock                                    # list configs and commands
+api-dock init [--force]                     # create api_dock_config/
+api-dock start [config_name]                # serve api_dock_config/<config_name>.yaml (default: config)
+api-dock start --backbone flask --host 127.0.0.1 --port 9000 --log-level debug
+api-dock describe [config_name]             # print the config
+api-dock generate-key                       # local encryption key
+api-dock encrypt "secret"                   # also --method env_key|aws_kms
+api-dock decrypt "gAAAAA..."
+api-dock lookups                            # run the config's lookups and print their rows
+```
+
+Flask responses are buffered and Flask refuses configs with PostgreSQL connections; use the default
+FastAPI backbone for those. Full reference: [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started#cli-reference).
+
+---
+
+## How it works (in brief)
+
+- **Configs.** The main `config.yaml` lists remotes and databases and is read at startup. Remote
+  files, database files and the shared `remotes/config.yaml` and `databases/config.yaml` are read
+  again on each request, so
+  route edits don't need a restart.
+- **Remotes.** A request is checked against the remote's allow/block lists, then forwarded with
+  httpx. The FastAPI app streams the upstream response back.
+- **Databases.** The version is resolved (`latest`, version files, `slugs`), the route matched and
+  its SQL built: `[[table]]` references expanded, query-param fragments appended, values bound.
+- **Engines.** A route whose tables are all on one PostgreSQL connection runs natively through
+  that connection's pool. Anything else runs on an in-memory DuckDB in a worker thread, with
+  PostgreSQL attached read-only when needed.
+- **Lookups.** Named queries (SQL over the configured tables, or an HTTP API) run at startup
+  and on a schedule; `from:` entries turn their rows into database versions or remote versions.
+- **Startup checks.** Every database and version is checked as requests will see it (table
+  references, quoted variables, unions, connections, engines); a bad config stops startup with a
+  message naming the database, version and route.
+
+---
+
+## Repo layout
+
+```
+api_dock/                   the package
+  cli.py                    api-dock commands
+  config.py, config_discovery.py   main/remote config loading, settings, route rules
+  route_mapper.py           RouteMapper: request handling, startup checks, PostgreSQL lifecycle
+  fast_api.py, flask_api.py the two app backbones
+  database_config.py        database configs, shared config, versions and slugs
+  lookups.py                lookups: templates, SQL/HTTP runners, refresh
+  sql_builder.py            SQL building, table references, unions, engine choice
+  database_backends.py      DuckDB backend    postgres_backend.py, postgres_config.py   PostgreSQL
+  storage_auth.py, auth.py, encryption.py, listings.py, sql_template_check.py, types.py
+  example_api_dock_config/  copied by `api-dock init`
+tests/                      pixi run -e dev pytest -q
+```
+
+More in the [Developer Guide](https://github.com/SchmidtDSE/api_dock/wiki/Developer-Guide).
+
+---
+
+# Development
+
+```bash
+pixi install -e dev           # includes psycopg and a local PostgreSQL for the tests
+pixi run -e dev pytest -q     # PostgreSQL tests skip if PostgreSQL/psycopg are missing
+```
+
+## Publishing a Release
+
+Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publish_to_pypi.yml` runs on `release: published`, builds the sdist and wheel with `uv build`, and uploads them using PyPI trusted publishing (OIDC). There's no local build, no API token, and no `twine`. conda-forge follows automatically: its bot opens a version PR on [conda-forge/api_dock-feedstock](https://github.com/conda-forge/api_dock-feedstock), which a maintainer merges.
+
+```bash
+# 0. Start from a clean, up-to-date main
+export VERSION=0.10.0         # the NEW version, no leading "v"
+git checkout main
+git pull origin main
+git status
+
+# 1. Set `version` in pyproject.toml to $VERSION
+
+# 2. Run the tests
+pixi run -e dev pytest -q
+
+# 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
+export COMMIT_MESSAGE='code review fixes: remote cookies, safer SQL building, auth caching'
+git add -A
+git commit -m "v$VERSION: $COMMIT_MESSAGE"
+git tag "v$VERSION"
+git push origin main "v$VERSION"
+
+# 4. Publish the GitHub Release; this triggers the PyPI upload.
+#    Don't use --draft (the workflow only runs on a published release); no wheel needs attaching.
+gh release create "v$VERSION" \
+  --title "v$VERSION" \
+  --notes "$(cat <<'EOF'
+**Upgrading:** a remote now receives only the cookies its `cookies:` setting allows (none without it). A remote that relied on the browser's cookies being passed through needs `cookies: [<name>]`, or `cookies: true` for all of them. Configs that set `action`, an unknown version `schema:`, non-boolean `add_trailing_slash`/`follow_redirects`, or a bad `timeout` now stop at startup.
+
+* new features
+    - `api-dock describe` builds the API like `start` (lookups, startup checks) and prints every database/version with expanded route SQL, plus each remote's url
+    - `api-dock init --force` replaces existing files; the whole example tree is copied
+    - `gcp` extra (`pip install 'api_dock[gcp]'`) for GCP Secret Manager authentication
+    - Startup warning when an `authentication` block would be ignored by remote routes
+* bug fixes
+    - Remote cookies: the client's raw `Cookie` header is no longer forwarded; the upstream `Cookie` header is built from the remote's `cookies` setting, so filtering and injected cookies work even when the client sends cookies
+    - Several upstream `Set-Cookie` headers reach the client separately instead of merged into one (new `ProxyResponse.set_cookies`)
+    - Query-param filters join the base query's top-level `WHERE` (strings, comments, CTEs and subqueries ignored), parenthesized and before a trailing `GROUP BY`/`ORDER BY`/`LIMIT`: fixes `WHERE a OR b` letting rows past filters, CTE-only `WHERE`s and base queries ending in `ORDER BY`
+    - `[[table]]` is a table source only right after `FROM`/`JOIN` or a `FROM`-list comma (was any `FROM`/`JOIN` in the previous 20 characters, which broke `ON [[a]].id = [[b]].id`)
+    - `sql_append` values are limited to integers and column names (with `ASC`/`DESC`, `NULLS FIRST/LAST`); the unimplemented `action` (which echoed request values, including cookies) is refused
+    - Authentication providers are built once per config instead of on every request (no secret-store/KMS call per request); `refresh_interval` now refreshes, keeping the last values if a refresh fails; tokens compared in constant time; `aws_tokens_file` works (`aws_key_id` optional)
+    - Remote route mapping fills `{{route_name}}` and `{{cookies.x}}` in `remote_route`, and keeps a query string written there
+    - Include/exclude and listing filters compare versions part by part (`1.10` no longer matches `1.1`)
+    - Values in DuckDB secret SQL (S3 region, GCS keys, HTTP headers) are quoted; a second GCS `service_account` no longer overwrites the process-wide one
+    - A main config that is missing (explicit path), invalid YAML or not a mapping stops startup instead of serving an empty API; settings are validated at startup
+    - An explicit encryption `key_file` must exist (only the default key file falls back to `API_DOCK_ENCRYPTION_KEY`); `env_key` reads only its variable
+    - Remote error responses no longer include internal error text (it's logged); `map_route_sync` no longer leaks event loops
+* cleanup / other improvements
+    - `map_database_route` split into named steps; one version-resolution helper for remotes and databases
+    - A remote request reads only that remote's config file (was every remote file, twice)
+    - Removed unused internal functions; shared YAML-loading and version-matching helpers; `follow_protocol_downgrades` (never implemented) removed
+    - FastAPI app reports the package version; classifiers match Python 3.11+; style fixes
+    - Tests grew from 626 to 760, including the first tests for authentication, encryption, config discovery and storage credentials
+EOF
+)"
+
+# 5. Watch the publish workflow, then confirm PyPI has the new version
+gh run watch "$(gh run list --workflow=publish_to_pypi.yml -L1 --json databaseId -q '.[0].databaseId')" --repo SchmidtDSE/api_dock
+curl -s https://pypi.org/pypi/api-dock/json | python3 -c "import sys,json; print('PyPI latest:', json.load(sys.stdin)['info']['version'])"
+
+# 6. conda-forge: once the bot opens the v$VERSION PR (usually within hours), check that the recipe's
+#    run requirements match pyproject.toml dependencies (the bot only bumps version + sha256), then merge it
+gh pr list --repo conda-forge/api_dock-feedstock --state open
+```
+
+---
+
+# License
+
+BSD 3-Clause

api_dock-0.10.0/README.md

@@ -0,0 +1,304 @@
+# API Dock
+
+API Dock builds an API from YAML files. Behind one endpoint it proxies requests to **remote HTTP
+APIs** and serves **SQL routes** over Parquet/CSV files (local disk, S3, GCS, Azure or HTTPS) and
+**PostgreSQL** tables. Run it with the `api-dock` CLI as a FastAPI or Flask app, or embed its
+`RouteMapper` in an existing Python service.
+
+| source | served at | how |
+|---|---|---|
+| remote HTTP API | `/<remote>/<path>` | forwarded upstream and streamed back, with optional allow-lists and restrictions |
+| file tables (Parquet, CSV, ...) | `/<database>/<version>/<route>` | SQL run by DuckDB |
+| PostgreSQL tables | `/<database>/<version>/<route>` | SQL run natively, or by DuckDB when a route mixes sources |
+
+Request values reach the database as bound parameters, never pasted into SQL, and every database
+config is checked when the server starts.
+
+**Documentation lives in the [wiki](https://github.com/SchmidtDSE/api_dock/wiki/).** Start with [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) and
+[Concepts](https://github.com/SchmidtDSE/api_dock/wiki/Concepts).
+
+---
+
+## Install
+
+```bash
+pip install api-dock                     # core
+pip install 'api-dock[postgres]'         # adds the PostgreSQL driver and pool
+conda install -c conda-forge api_dock    # or from conda-forge
+```
+
+API Dock needs Python 3.11 or later.
+
+---
+
+## Quick start
+
+```bash
+api-dock init        # creates api_dock_config/ with commented example files
+```
+
+Replace the examples with a main config, one remote and one database:
+
+```
+api_dock_config/
+├── config.yaml
+├── remotes/
+│   └── httpbin.yaml
+└── databases/
+    └── places.yaml
+```
+
+```yaml
+# api_dock_config/config.yaml
+name: my-api
+description: My first API Dock
+remotes:
+  - httpbin
+databases:
+  - places
+settings:
+  add_trailing_slash: false   # httpbin doesn't want /get/
+```
+
+```yaml
+# api_dock_config/remotes/httpbin.yaml
+name: httpbin
+url: https://httpbin.org
+```
+
+```yaml
+# api_dock_config/databases/places.yaml
+tables:
+  places: data/places.parquet            # or s3://bucket/places/**/*.parquet, a PostgreSQL table, ...
+routes:
+  - route: places
+    sql: SELECT * FROM [[places]]
+    query_params:
+      - country:
+          sql: "country = {{country}}"
+  - route: places/{{id}}
+    sql: SELECT * FROM [[places]] WHERE id = {{id}}
+```
+
+`[[places]]` is replaced by the table's source; `{{id}}` and `{{country}}` are request values,
+sent to the database as bound parameters.
+
+```bash
+api-dock start                                    # FastAPI on port 8000
+curl http://localhost:8000/httpbin/get            # proxied to https://httpbin.org/get
+curl http://localhost:8000/places/places?country=FR
+# [{"id": 2, "name": "Lyon", "country": "FR"}]
+curl http://localhost:8000/places/places/1
+```
+
+[Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started) walks through this example, including making the Parquet
+file.
+
+---
+
+## What you can configure
+
+| topic | wiki page |
+|---|---|
+| main config, settings (`timeout`, `base_path`, `duckdb`, ...), multiple configs | [Configuration](https://github.com/SchmidtDSE/api_dock/wiki/Configuration) |
+| versioned remotes and databases, inline versions, `latest` | [Versioning](https://github.com/SchmidtDSE/api_dock/wiki/Versioning) |
+| remote configs (files or `remotes/config.yaml`), allow-lists, `restricted` patterns, route mapping | [Routing and Restrictions](https://github.com/SchmidtDSE/api_dock/wiki/Routing-and-Restrictions) |
+| cookies, database authentication, encrypted values | [Authentication and Cookies](https://github.com/SchmidtDSE/api_dock/wiki/Authentication-and-Cookies) |
+| tables, `[[table]]` references, routes, startup checks | [SQL Database Support](https://github.com/SchmidtDSE/api_dock/wiki/SQL-Database-Support) |
+| filtering, sorting, pagination, required params | [Query Parameters](https://github.com/SchmidtDSE/api_dock/wiki/Query-Parameters) |
+| picking SQL from the request | [Conditional SQL](https://github.com/SchmidtDSE/api_dock/wiki/Conditional-SQL) |
+| shared tables, schemas, slugs, shared routes | [Shared Database Config](https://github.com/SchmidtDSE/api_dock/wiki/Shared-Database-Config) |
+| unions across schemas (`[[*.table]]`, schema groups) | [Cross-Schema Queries](https://github.com/SchmidtDSE/api_dock/wiki/Cross-Schema-Queries) |
+| PostgreSQL connections, engines, safety | [PostgreSQL](https://github.com/SchmidtDSE/api_dock/wiki/PostgreSQL) |
+| databases, versions and values generated from a query, refreshed on a schedule | [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups) |
+| `/databases`, `/remotes`, `/sources` listings | [Catalog Endpoints](https://github.com/SchmidtDSE/api_dock/wiki/Catalog-Endpoints) |
+| `RouteMapper` in your own app, production deployment | [Python API and Deployment](https://github.com/SchmidtDSE/api_dock/wiki/Python-API-and-Deployment) |
+
+---
+
+## Example: versions from a catalog (lookups)
+
+When the list of databases/versions lives somewhere else (a table of model runs, a
+deployments API), a lookup turns its rows into config. api_dock runs it at startup, every
+`refresh` and on demand:
+
+```yaml
+# api_dock_config/databases/config.yaml
+database:
+  connections:
+    core: {host: db.example.com, dbname: catalog, user: readonly, password: env:DB_PASSWORD}
+  runs_catalog: {connection: core, table: public.model_runs}   # or {uri: s3://.../runs.parquet}
+
+lookups:
+  model_runs:
+    sql: |
+      SELECT name, version, detections_uri,
+             replace(name, '-', '_') || '_' || replace(version, '.', 'p') AS schema
+      FROM [[runs_catalog]] WHERE published
+    refresh: 7d
+    allow: ["s3://my-bucket/runs/"]        # URIs from rows must start with this
+
+slugs:
+  - from: model_runs                       # one database/version per row
+    name: "{{row.name}}"
+    version: "{{row.version}}"
+    schema:
+      name: "{{row.schema}}"
+      tables:
+        detections: {uri: "{{row.detections_uri}}"}
+```
+
+```yaml
+# api_dock_config/config.yaml
+databases:
+  - from: model_runs                       # serve every database the lookup generates
+settings:
+  lookups:                                 # optional: GET status / POST refresh
+    refresh_route: /admin/lookups
+    token: env:API_DOCK_ADMIN_TOKEN
+```
+
+Lookups can read PostgreSQL tables, Parquet/CSV files or an HTTP API, and can also
+generate remote versions or fill single values. See [Lookups](https://github.com/SchmidtDSE/api_dock/wiki/Lookups).
+
+---
+
+## CLI
+
+```bash
+api-dock                                    # list configs and commands
+api-dock init [--force]                     # create api_dock_config/
+api-dock start [config_name]                # serve api_dock_config/<config_name>.yaml (default: config)
+api-dock start --backbone flask --host 127.0.0.1 --port 9000 --log-level debug
+api-dock describe [config_name]             # print the config
+api-dock generate-key                       # local encryption key
+api-dock encrypt "secret"                   # also --method env_key|aws_kms
+api-dock decrypt "gAAAAA..."
+api-dock lookups                            # run the config's lookups and print their rows
+```
+
+Flask responses are buffered and Flask refuses configs with PostgreSQL connections; use the default
+FastAPI backbone for those. Full reference: [Getting Started](https://github.com/SchmidtDSE/api_dock/wiki/Getting-Started#cli-reference).
+
+---
+
+## How it works (in brief)
+
+- **Configs.** The main `config.yaml` lists remotes and databases and is read at startup. Remote
+  files, database files and the shared `remotes/config.yaml` and `databases/config.yaml` are read
+  again on each request, so
+  route edits don't need a restart.
+- **Remotes.** A request is checked against the remote's allow/block lists, then forwarded with
+  httpx. The FastAPI app streams the upstream response back.
+- **Databases.** The version is resolved (`latest`, version files, `slugs`), the route matched and
+  its SQL built: `[[table]]` references expanded, query-param fragments appended, values bound.
+- **Engines.** A route whose tables are all on one PostgreSQL connection runs natively through
+  that connection's pool. Anything else runs on an in-memory DuckDB in a worker thread, with
+  PostgreSQL attached read-only when needed.
+- **Lookups.** Named queries (SQL over the configured tables, or an HTTP API) run at startup
+  and on a schedule; `from:` entries turn their rows into database versions or remote versions.
+- **Startup checks.** Every database and version is checked as requests will see it (table
+  references, quoted variables, unions, connections, engines); a bad config stops startup with a
+  message naming the database, version and route.
+
+---
+
+## Repo layout
+
+```
+api_dock/                   the package
+  cli.py                    api-dock commands
+  config.py, config_discovery.py   main/remote config loading, settings, route rules
+  route_mapper.py           RouteMapper: request handling, startup checks, PostgreSQL lifecycle
+  fast_api.py, flask_api.py the two app backbones
+  database_config.py        database configs, shared config, versions and slugs
+  lookups.py                lookups: templates, SQL/HTTP runners, refresh
+  sql_builder.py            SQL building, table references, unions, engine choice
+  database_backends.py      DuckDB backend    postgres_backend.py, postgres_config.py   PostgreSQL
+  storage_auth.py, auth.py, encryption.py, listings.py, sql_template_check.py, types.py
+  example_api_dock_config/  copied by `api-dock init`
+tests/                      pixi run -e dev pytest -q
+```
+
+More in the [Developer Guide](https://github.com/SchmidtDSE/api_dock/wiki/Developer-Guide).
+
+---
+
+# Development
+
+```bash
+pixi install -e dev           # includes psycopg and a local PostgreSQL for the tests
+pixi run -e dev pytest -q     # PostgreSQL tests skip if PostgreSQL/psycopg are missing
+```
+
+## Publishing a Release
+
+Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publish_to_pypi.yml` runs on `release: published`, builds the sdist and wheel with `uv build`, and uploads them using PyPI trusted publishing (OIDC). There's no local build, no API token, and no `twine`. conda-forge follows automatically: its bot opens a version PR on [conda-forge/api_dock-feedstock](https://github.com/conda-forge/api_dock-feedstock), which a maintainer merges.
+
+```bash
+# 0. Start from a clean, up-to-date main
+export VERSION=0.10.0         # the NEW version, no leading "v"
+git checkout main
+git pull origin main
+git status
+
+# 1. Set `version` in pyproject.toml to $VERSION
+
+# 2. Run the tests
+pixi run -e dev pytest -q
+
+# 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
+export COMMIT_MESSAGE='code review fixes: remote cookies, safer SQL building, auth caching'
+git add -A
+git commit -m "v$VERSION: $COMMIT_MESSAGE"
+git tag "v$VERSION"
+git push origin main "v$VERSION"
+
+# 4. Publish the GitHub Release; this triggers the PyPI upload.
+#    Don't use --draft (the workflow only runs on a published release); no wheel needs attaching.
+gh release create "v$VERSION" \
+  --title "v$VERSION" \
+  --notes "$(cat <<'EOF'
+**Upgrading:** a remote now receives only the cookies its `cookies:` setting allows (none without it). A remote that relied on the browser's cookies being passed through needs `cookies: [<name>]`, or `cookies: true` for all of them. Configs that set `action`, an unknown version `schema:`, non-boolean `add_trailing_slash`/`follow_redirects`, or a bad `timeout` now stop at startup.
+
+* new features
+    - `api-dock describe` builds the API like `start` (lookups, startup checks) and prints every database/version with expanded route SQL, plus each remote's url
+    - `api-dock init --force` replaces existing files; the whole example tree is copied
+    - `gcp` extra (`pip install 'api_dock[gcp]'`) for GCP Secret Manager authentication
+    - Startup warning when an `authentication` block would be ignored by remote routes
+* bug fixes
+    - Remote cookies: the client's raw `Cookie` header is no longer forwarded; the upstream `Cookie` header is built from the remote's `cookies` setting, so filtering and injected cookies work even when the client sends cookies
+    - Several upstream `Set-Cookie` headers reach the client separately instead of merged into one (new `ProxyResponse.set_cookies`)
+    - Query-param filters join the base query's top-level `WHERE` (strings, comments, CTEs and subqueries ignored), parenthesized and before a trailing `GROUP BY`/`ORDER BY`/`LIMIT`: fixes `WHERE a OR b` letting rows past filters, CTE-only `WHERE`s and base queries ending in `ORDER BY`
+    - `[[table]]` is a table source only right after `FROM`/`JOIN` or a `FROM`-list comma (was any `FROM`/`JOIN` in the previous 20 characters, which broke `ON [[a]].id = [[b]].id`)
+    - `sql_append` values are limited to integers and column names (with `ASC`/`DESC`, `NULLS FIRST/LAST`); the unimplemented `action` (which echoed request values, including cookies) is refused
+    - Authentication providers are built once per config instead of on every request (no secret-store/KMS call per request); `refresh_interval` now refreshes, keeping the last values if a refresh fails; tokens compared in constant time; `aws_tokens_file` works (`aws_key_id` optional)
+    - Remote route mapping fills `{{route_name}}` and `{{cookies.x}}` in `remote_route`, and keeps a query string written there
+    - Include/exclude and listing filters compare versions part by part (`1.10` no longer matches `1.1`)
+    - Values in DuckDB secret SQL (S3 region, GCS keys, HTTP headers) are quoted; a second GCS `service_account` no longer overwrites the process-wide one
+    - A main config that is missing (explicit path), invalid YAML or not a mapping stops startup instead of serving an empty API; settings are validated at startup
+    - An explicit encryption `key_file` must exist (only the default key file falls back to `API_DOCK_ENCRYPTION_KEY`); `env_key` reads only its variable
+    - Remote error responses no longer include internal error text (it's logged); `map_route_sync` no longer leaks event loops
+* cleanup / other improvements
+    - `map_database_route` split into named steps; one version-resolution helper for remotes and databases
+    - A remote request reads only that remote's config file (was every remote file, twice)
+    - Removed unused internal functions; shared YAML-loading and version-matching helpers; `follow_protocol_downgrades` (never implemented) removed
+    - FastAPI app reports the package version; classifiers match Python 3.11+; style fixes
+    - Tests grew from 626 to 760, including the first tests for authentication, encryption, config discovery and storage credentials
+EOF
+)"
+
+# 5. Watch the publish workflow, then confirm PyPI has the new version
+gh run watch "$(gh run list --workflow=publish_to_pypi.yml -L1 --json databaseId -q '.[0].databaseId')" --repo SchmidtDSE/api_dock
+curl -s https://pypi.org/pypi/api-dock/json | python3 -c "import sys,json; print('PyPI latest:', json.load(sys.stdin)['info']['version'])"
+
+# 6. conda-forge: once the bot opens the v$VERSION PR (usually within hours), check that the recipe's
+#    run requirements match pyproject.toml dependencies (the bot only bumps version + sha256), then merge it
+gh pr list --repo conda-forge/api_dock-feedstock --state open
+```
+
+---
+
+# License
+
+BSD 3-Clause