api-dock 0.8.1__tar.gz → 0.8.2__tar.gz

{api_dock-0.8.1 → api_dock-0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: api_dock
-Version: 0.8.1
+Version: 0.8.2
 Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
 Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
 License-Expression: BSD-3-Clause
@@ -219,19 +219,33 @@ remotes:
 settings:
   add_trailing_slash: true              # Auto-add trailing slash to paths (default: true)
   follow_protocol_downgrades: false     # Allow HTTPS->HTTP redirects (default: false)
+  follow_redirects: true                # Follow remote redirects (default: true)
   timeout: 10                           # Upstream request timeout in seconds (default: 10)
+  base_path: /dock                      # Also serve the API under this prefix (default: none)
+  duckdb:                               # Options for database queries (default: none)
+    memory_limit: 700MB
+    threads: 2
+    max_concurrent_queries: 2
 ```
 
-### HTTP behavior Settings
+### Settings
 
-The optional `settings` section controls HTTP behavior:
+The optional `settings` section controls HTTP and query behavior:
 
 - **`add_trailing_slash`** (default: `true`): Automatically append a trailing slash to all proxied paths. This prevents 307/301 redirects from remote APIs that require trailing slashes (e.g., `/projects` → `/projects/`). Set to `false` to disable this behavior.
 
 - **`follow_protocol_downgrades`** (default: `false`): Control how HTTP redirects are handled. When `false` (recommended), HTTPS→HTTP redirects are blocked for security. When `true`, allows following redirects that downgrade from HTTPS to HTTP (not recommended for production).
 
+- **`follow_redirects`** (default: `true`): Whether redirects from a remote are followed by API Dock (`true`) or passed through to the client with their `Location` header (`false`). Set it to `false` when a remote answers with redirects the client should follow itself, such as presigned S3 URLs for large files.
+
 - **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
 
+- **`base_path`** (default: none): An extra URL prefix the API is also served under, e.g. `/dock`. Use it when a proxy or CDN forwards a path on another domain without stripping it (say CloudFront routes `https://app.example.org/dock/*` to API Dock): `/dock/birdnet/latest/detections/` is then handled as `/birdnet/latest/detections/`. Paths without the prefix keep working, so direct calls and health checks are unaffected.
+
+- **`duckdb`** (default: none): Options for the DuckDB connection each database query runs on. Every key except `max_concurrent_queries` is applied as `SET <key> = <value>`, so any [DuckDB setting](https://duckdb.org/docs/configuration/overview) works; the useful ones on small servers are `memory_limit` (DuckDB spills to disk or fails the query instead of exceeding it), `threads`, and `temp_directory`. `max_concurrent_queries` caps how many database queries run at once in the process; further queries wait their turn. Memory limits apply per query, so on a small instance set `memory_limit × max_concurrent_queries` below the instance's memory.
+
+Database queries run in worker threads, so a slow query doesn't hold up other requests (including health checks on `/`).
+
 ### Catalog Endpoints (`expose`)
 
 The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
@@ -1601,7 +1615,7 @@ Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publis
 
 ```bash
 # 0. Start from a clean, up-to-date main
-export VERSION=0.8.1          # the NEW version, no leading "v"
+export VERSION=0.8.2          # the NEW version, no leading "v"
 git checkout main
 git pull origin main
 git status
@@ -1612,7 +1626,7 @@ git status
 pixi run -e dev pytest -q
 
 # 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
-export COMMIT_MESSAGE='cross-schema unions, schema groups, source columns'
+export COMMIT_MESSAGE='query worker threads, duckdb settings, base_path, proxy host fix'
 git add -A
 git commit -m "v$VERSION: $COMMIT_MESSAGE"
 git tag "v$VERSION"
@@ -1624,17 +1638,14 @@ gh release create "v$VERSION" \
   --title "v$VERSION" \
   --notes "$(cat <<'EOF'
 * new features
-    - Cross-schema unions: `[[*.table]]` reads a table from every shared schema that has it, and `[[*!.table]]` does the same minus the current database/version's schema
-    - Named `schema_groups` in `databases/config.yaml`, used as `[[group.table]]` / `[[group!.table]]` (validated: known schemas only, no group/schema name clashes)
-    - Route `source_columns` adds where each union row came from (`schema`, `name`, `version`), with default names `schema_name`/`name`/`version` or your own; nothing is added by default
-    - `{{self.schema}}`, `{{self.name}}`, and `{{self.version}}` placeholders for the database/version being queried
-    - Together these support an "overlaps" route shared by every database/version (every detection overlapping a given one across all schemas, except that detection itself); shared query params such as `confidence`, `sort`, and `limit` apply to its rows
+    - `settings.duckdb`: DuckDB options applied to every database query (`memory_limit`, `threads`, `temp_directory`, or any other DuckDB setting), plus `max_concurrent_queries` to cap how many queries run at once
+    - `settings.base_path`: also serve the API under a URL prefix (e.g. `/dock`), for a CDN/proxy path such as CloudFront routing `https://app.example.org/dock/*` to API Dock; unprefixed paths keep working
 * bug fixes
-    - A `!` union that removes every member returns no rows (with the right columns) instead of failing, and only the schemas a union actually reads get views and storage credentials
+    - Database queries now run in worker threads, so one slow query no longer stalls every other request (including health checks on `/`)
+    - Remote proxying no longer forwards the client's `Host` (or hop-by-hop) headers upstream; upstream redirects (e.g. trailing-slash 307s) no longer point back at the proxy with the wrong host and path
 * cleanup / other improvements
-    - Added `SqlContext`; `build_sql_query()` / `build_sql_query_with_tables()` accept an optional `context`
-    - README: new "Querying across schemas" section with an overlaps example; example `databases/config.yaml` shows `schema_groups` and a union route
-    - Test suite grew from 198 to 227 tests (`test_schema_unions.py`, including a real DuckDB end-to-end overlaps test)
+    - README: document `base_path`, `duckdb`, and the previously undocumented `follow_redirects` setting
+    - Test suite grew from 227 to 253 tests (`test_runtime_settings.py`)
 EOF
 )"
 

{api_dock-0.8.1 → api_dock-0.8.2}/README.md

@@ -180,19 +180,33 @@ remotes:
 settings:
   add_trailing_slash: true              # Auto-add trailing slash to paths (default: true)
   follow_protocol_downgrades: false     # Allow HTTPS->HTTP redirects (default: false)
+  follow_redirects: true                # Follow remote redirects (default: true)
   timeout: 10                           # Upstream request timeout in seconds (default: 10)
+  base_path: /dock                      # Also serve the API under this prefix (default: none)
+  duckdb:                               # Options for database queries (default: none)
+    memory_limit: 700MB
+    threads: 2
+    max_concurrent_queries: 2
 ```
 
-### HTTP behavior Settings
+### Settings
 
-The optional `settings` section controls HTTP behavior:
+The optional `settings` section controls HTTP and query behavior:
 
 - **`add_trailing_slash`** (default: `true`): Automatically append a trailing slash to all proxied paths. This prevents 307/301 redirects from remote APIs that require trailing slashes (e.g., `/projects` → `/projects/`). Set to `false` to disable this behavior.
 
 - **`follow_protocol_downgrades`** (default: `false`): Control how HTTP redirects are handled. When `false` (recommended), HTTPS→HTTP redirects are blocked for security. When `true`, allows following redirects that downgrade from HTTPS to HTTP (not recommended for production).
 
+- **`follow_redirects`** (default: `true`): Whether redirects from a remote are followed by API Dock (`true`) or passed through to the client with their `Location` header (`false`). Set it to `false` when a remote answers with redirects the client should follow itself, such as presigned S3 URLs for large files.
+
 - **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
 
+- **`base_path`** (default: none): An extra URL prefix the API is also served under, e.g. `/dock`. Use it when a proxy or CDN forwards a path on another domain without stripping it (say CloudFront routes `https://app.example.org/dock/*` to API Dock): `/dock/birdnet/latest/detections/` is then handled as `/birdnet/latest/detections/`. Paths without the prefix keep working, so direct calls and health checks are unaffected.
+
+- **`duckdb`** (default: none): Options for the DuckDB connection each database query runs on. Every key except `max_concurrent_queries` is applied as `SET <key> = <value>`, so any [DuckDB setting](https://duckdb.org/docs/configuration/overview) works; the useful ones on small servers are `memory_limit` (DuckDB spills to disk or fails the query instead of exceeding it), `threads`, and `temp_directory`. `max_concurrent_queries` caps how many database queries run at once in the process; further queries wait their turn. Memory limits apply per query, so on a small instance set `memory_limit × max_concurrent_queries` below the instance's memory.
+
+Database queries run in worker threads, so a slow query doesn't hold up other requests (including health checks on `/`).
+
 ### Catalog Endpoints (`expose`)
 
 The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
@@ -1562,7 +1576,7 @@ Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publis
 
 ```bash
 # 0. Start from a clean, up-to-date main
-export VERSION=0.8.1          # the NEW version, no leading "v"
+export VERSION=0.8.2          # the NEW version, no leading "v"
 git checkout main
 git pull origin main
 git status
@@ -1573,7 +1587,7 @@ git status
 pixi run -e dev pytest -q
 
 # 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
-export COMMIT_MESSAGE='cross-schema unions, schema groups, source columns'
+export COMMIT_MESSAGE='query worker threads, duckdb settings, base_path, proxy host fix'
 git add -A
 git commit -m "v$VERSION: $COMMIT_MESSAGE"
 git tag "v$VERSION"
@@ -1585,17 +1599,14 @@ gh release create "v$VERSION" \
   --title "v$VERSION" \
   --notes "$(cat <<'EOF'
 * new features
-    - Cross-schema unions: `[[*.table]]` reads a table from every shared schema that has it, and `[[*!.table]]` does the same minus the current database/version's schema
-    - Named `schema_groups` in `databases/config.yaml`, used as `[[group.table]]` / `[[group!.table]]` (validated: known schemas only, no group/schema name clashes)
-    - Route `source_columns` adds where each union row came from (`schema`, `name`, `version`), with default names `schema_name`/`name`/`version` or your own; nothing is added by default
-    - `{{self.schema}}`, `{{self.name}}`, and `{{self.version}}` placeholders for the database/version being queried
-    - Together these support an "overlaps" route shared by every database/version (every detection overlapping a given one across all schemas, except that detection itself); shared query params such as `confidence`, `sort`, and `limit` apply to its rows
+    - `settings.duckdb`: DuckDB options applied to every database query (`memory_limit`, `threads`, `temp_directory`, or any other DuckDB setting), plus `max_concurrent_queries` to cap how many queries run at once
+    - `settings.base_path`: also serve the API under a URL prefix (e.g. `/dock`), for a CDN/proxy path such as CloudFront routing `https://app.example.org/dock/*` to API Dock; unprefixed paths keep working
 * bug fixes
-    - A `!` union that removes every member returns no rows (with the right columns) instead of failing, and only the schemas a union actually reads get views and storage credentials
+    - Database queries now run in worker threads, so one slow query no longer stalls every other request (including health checks on `/`)
+    - Remote proxying no longer forwards the client's `Host` (or hop-by-hop) headers upstream; upstream redirects (e.g. trailing-slash 307s) no longer point back at the proxy with the wrong host and path
 * cleanup / other improvements
-    - Added `SqlContext`; `build_sql_query()` / `build_sql_query_with_tables()` accept an optional `context`
-    - README: new "Querying across schemas" section with an overlaps example; example `databases/config.yaml` shows `schema_groups` and a union route
-    - Test suite grew from 198 to 227 tests (`test_schema_unions.py`, including a real DuckDB end-to-end overlaps test)
+    - README: document `base_path`, `duckdb`, and the previously undocumented `follow_redirects` setting
+    - Test suite grew from 227 to 253 tests (`test_runtime_settings.py`)
 EOF
 )"
 

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock/example_api_dock_config/config.yaml

@@ -15,6 +15,11 @@ settings:
   add_trailing_slash: false   # Set true to auto-append trailing slash to proxied paths
   follow_redirects: true      # Set false to pass 3xx redirects through to the client
   timeout: 10                 # Upstream request timeout (seconds); null/false to disable
+  # base_path: /dock          # Also serve the API under this prefix (e.g. behind a CDN path)
+  # duckdb:                   # Database query options (DuckDB SET options + a concurrency cap)
+  #   memory_limit: 700MB
+  #   threads: 2
+  #   max_concurrent_queries: 2
 
 # Global route restrictions — applied to all remotes unless overridden per-remote.
 # Uncomment to block DELETE on every remote:

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock/fast_api.py

@@ -16,9 +16,14 @@ import warnings
 import httpx
 from fastapi import FastAPI, Request
 from fastapi.responses import JSONResponse, Response, StreamingResponse
-from typing import Any, Dict, Optional
-
-from api_dock.route_mapper import collect_multi_query_params, HOP_BY_HOP_HEADERS, RouteMapper
+from typing import Any, Callable, Dict, Optional
+
+from api_dock.route_mapper import (
+    collect_multi_query_params,
+    HOP_BY_HOP_HEADERS,
+    RouteMapper,
+    strip_base_path,
+)
 from api_dock.types import PreparedRequest, ProxyResponse
 
 
@@ -66,12 +71,36 @@ def create_app(config_path: Optional[str] = None) -> FastAPI:
     _add_remote_routes(app, route_mapper)
     _add_error_handlers(app)
 
+    if route_mapper.base_path:
+        app.add_middleware(_StripBasePath, base_path=route_mapper.base_path)
+
     return app
 
 
 #
 # INTERNAL
 #
+class _StripBasePath:
+    """ASGI middleware serving the app under ``settings.base_path`` as well.
+
+    Requests whose path starts with the base path (e.g. ``/dock/birdnet/...``)
+    are routed as if it weren't there; other paths pass through unchanged.
+    """
+
+    def __init__(self, app: Callable, base_path: str) -> None:
+        self.app = app
+        self.base_path = base_path
+
+    async def __call__(self, scope: Dict[str, Any], receive: Callable, send: Callable) -> None:
+        """Rewrite the request path, then call the wrapped app."""
+        if scope.get("type") in ("http", "websocket"):
+            path = strip_base_path(scope.get("path", ""), self.base_path)
+            if path != scope.get("path"):
+                scope = {**scope, "path": path, "raw_path": path.encode()}
+        await self.app(scope, receive, send)
+
+
+
 def _add_main_routes(app: FastAPI, route_mapper: RouteMapper) -> None:
     """Add main API routes to the FastAPI app.
 

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock/flask_api.py

@@ -14,9 +14,9 @@ License: BSD 3-Clause
 import asyncio
 import warnings
 from flask import Flask, jsonify, request, Response as FlaskResponse
-from typing import Any, Dict, Optional
+from typing import Any, Callable, Dict, Optional
 
-from api_dock.route_mapper import collect_multi_query_params, RouteMapper
+from api_dock.route_mapper import collect_multi_query_params, RouteMapper, strip_base_path
 
 
 #
@@ -52,12 +52,35 @@ def create_app(config_path: Optional[str] = None) -> Flask:
     _add_main_routes(app, route_mapper)
     _add_error_handlers(app)
 
+    if route_mapper.base_path:
+        app.wsgi_app = _strip_base_path(app.wsgi_app, route_mapper.base_path)
+
     return app
 
 
 #
 # INTERNAL
 #
+def _strip_base_path(wsgi_app: Callable, base_path: str) -> Callable:
+    """Wrap a WSGI app so it is also served under ``settings.base_path``.
+
+    Args:
+        wsgi_app: The Flask WSGI app.
+        base_path: Normalized prefix, e.g. "/dock".
+
+    Returns:
+        WSGI app that routes ``/dock/...`` as ``/...`` (other paths unchanged).
+    """
+    def middleware(environ: Dict[str, Any], start_response: Callable) -> Any:
+        """Strip the base path from PATH_INFO, then call the Flask app."""
+        path = environ.get("PATH_INFO", "")
+        stripped = strip_base_path(path, base_path)
+        if stripped != path:
+            environ["PATH_INFO"] = stripped
+        return wsgi_app(environ, start_response)
+    return middleware
+
+
 def _add_main_routes(app: Flask, route_mapper: RouteMapper) -> None:
     """Add main API routes to the Flask app.
 

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock/route_mapper.py

@@ -11,7 +11,10 @@ License: BSD 3-Clause
 #
 # IMPORTS
 #
+import asyncio
 import json
+import re
+import threading
 import httpx
 import yaml
 from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
@@ -54,9 +57,77 @@ HOP_BY_HOP_HEADERS: frozenset = frozenset({
 })
 
 
+# Request headers never forwarded to an upstream remote. Host must be the
+# upstream's own (httpx sets it): forwarding the client's Host makes the upstream
+# build redirects and absolute URLs that point back at the proxy. The rest are
+# hop-by-hop headers, or (content-length) recomputed by httpx from the body.
+EXCLUDED_REQUEST_HEADERS: frozenset = frozenset({
+    "connection",
+    "content-length",
+    "host",
+    "keep-alive",
+    "proxy-authorization",
+    "te",
+    "trailers",
+    "transfer-encoding",
+    "upgrade",
+})
+
+# `settings.duckdb` options. Every key is applied to each query's DuckDB
+# connection as `SET <key> = <value>` (memory_limit, threads, temp_directory,
+# ...), except max_concurrent_queries, which caps how many database queries run
+# at once in this process (others wait their turn).
+DUCKDB_SETTINGS_KEY: str = "duckdb"
+MAX_CONCURRENT_QUERIES_KEY: str = "max_concurrent_queries"
+
+# `settings.base_path`: an optional URL prefix (e.g. "/dock") the API is also
+# served under, for when a proxy/CDN forwards a path prefix unchanged.
+BASE_PATH_KEY: str = "base_path"
+
+DUCKDB_OPTION_PATTERN: re.Pattern = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+
+
 #
 # PUBLIC
 #
+def normalize_base_path(base_path: Any) -> Optional[str]:
+    """Normalize a ``base_path`` setting to ``/prefix`` form.
+
+    Args:
+        base_path: Configured value (e.g. "dock", "/dock/"), or None/empty.
+
+    Returns:
+        "/prefix" without a trailing slash, or None if no prefix is set.
+    """
+    if not base_path:
+        return None
+    stripped = str(base_path).strip().strip("/")
+    return f"/{stripped}" if stripped else None
+
+
+def strip_base_path(path: str, base_path: Optional[str]) -> str:
+    """Remove a base path prefix from a request path, if present.
+
+    Paths without the prefix are returned unchanged, so the API answers both
+    with and without it (e.g. direct calls and health checks still work).
+
+    Args:
+        path: Request path, e.g. "/dock/birdnet/latest/detections/".
+        base_path: Normalized prefix (see normalize_base_path), or None.
+
+    Returns:
+        The path without the prefix ("/birdnet/latest/detections/"), "/" for
+        the prefix itself, or the original path.
+    """
+    if not base_path:
+        return path
+    if path == base_path or path == f"{base_path}/":
+        return "/"
+    if path.startswith(f"{base_path}/"):
+        return path[len(base_path):]
+    return path
+
+
 def collect_multi_query_params(items: Iterable[Tuple[str, str]]) -> Dict[str, List[str]]:
     """Group repeated query string pairs into a name-to-value-list mapping.
 
@@ -100,6 +171,11 @@ class RouteMapper:
         self.database_names = get_database_names(self.config)
         self.settings = get_settings(self.config)
         self.listing_specs, self.listing_warnings = resolve_listing_specs(self.config)
+        self.base_path = normalize_base_path(self.settings.get(BASE_PATH_KEY))
+        self.duckdb_statements, max_queries = _duckdb_settings(
+            self.settings.get(DUCKDB_SETTINGS_KEY)
+        )
+        self._query_slots = threading.BoundedSemaphore(max_queries) if max_queries else None
 
     def get_config_metadata(self) -> Dict[str, Any]:
         """Get API metadata from configuration.
@@ -251,7 +327,7 @@ class RouteMapper:
         return PreparedRequest(
             url=full_url,
             method=method,
-            headers=headers or {},
+            headers=_filter_request_headers(headers or {}),
             params=filtered_query_params,
             cookies=filtered_cookies,
             body=body,
@@ -492,35 +568,22 @@ class RouteMapper:
         except (ValueError, yaml.YAMLError):
             return _error_response(500, "SQL query error")
 
-        try:
-            import duckdb
-
-            conn = duckdb.connect(database=':memory:')
-
-            # Authenticate every local table (as before) plus any shared tables
-            # the query references, then expose [[schema.table]] refs as views.
-            auth_tables = get_local_table_references(database_config, shared_config)
-            local_names = {table.sql_name for table in auth_tables}
-            auth_tables += [ref for ref in table_refs if ref.sql_name not in local_names]
-            setup_table_storage_authentication(conn, auth_tables)
-            for statement in build_schema_view_statements(table_refs):
-                conn.execute(statement)
-
-            result = conn.execute(sql_query).fetchall()
-            columns = [desc[0] for desc in conn.description] if conn.description else []
-            conn.close()
-
-            response_data = []
-            for row in result:
-                row_dict = {}
-                for col, val in zip(columns, row):
-                    row_dict[col] = _make_json_safe(val)
-                response_data.append(row_dict)
-
-            return _json_response(response_data)
+        # Authenticate every local table plus any shared tables the query
+        # references, then expose [[schema.table]] refs as views.
+        auth_tables = get_local_table_references(database_config, shared_config)
+        local_names = {table.sql_name for table in auth_tables}
+        auth_tables += [ref for ref in table_refs if ref.sql_name not in local_names]
 
+        try:
+            # DuckDB calls block; run them in a worker thread so one slow query
+            # doesn't stall every other request (and health checks) meanwhile.
+            response_data = await asyncio.to_thread(
+                self._run_query, sql_query, auth_tables,
+                build_schema_view_statements(table_refs),
+            )
         except Exception:
             return _error_response(500, "Database query error")
+        return _json_response(response_data)
 
     def is_remote_name(self, name: str) -> bool:
         """Check if a given name is a configured remote name.
@@ -598,6 +661,51 @@ class RouteMapper:
         except Exception as e:
             return _error_response(500, f"Sync wrapper error: {str(e)}")
 
+    def _run_query(
+            self,
+            sql_query: str,
+            auth_tables: List[Any],
+            view_statements: List[str]) -> List[Dict[str, Any]]:
+        """Execute a database query on a fresh DuckDB connection (blocking).
+
+        Applies the ``settings.duckdb`` options, storage authentication and
+        schema views, then runs the query. Honors max_concurrent_queries.
+
+        Args:
+            sql_query: The SQL to run.
+            auth_tables: TableReferences to set up storage authentication for.
+            view_statements: CREATE SCHEMA/VIEW statements to run first.
+
+        Returns:
+            Result rows as JSON-safe dicts.
+        """
+        import duckdb
+
+        # getattr: RouteMappers built without __init__ (e.g. in tests) have neither.
+        slots = getattr(self, '_query_slots', None)
+        if slots is not None:
+            slots.acquire()
+        try:
+            conn = duckdb.connect(database=':memory:')
+            try:
+                for statement in getattr(self, 'duckdb_statements', []):
+                    conn.execute(statement)
+                setup_table_storage_authentication(conn, auth_tables)
+                for statement in view_statements:
+                    conn.execute(statement)
+                result = conn.execute(sql_query).fetchall()
+                columns = [desc[0] for desc in conn.description] if conn.description else []
+            finally:
+                conn.close()
+        finally:
+            if slots is not None:
+                slots.release()
+
+        return [
+            {column: _make_json_safe(value) for column, value in zip(columns, row)}
+            for row in result
+        ]
+
     def _is_remote_filename(self, filename: str) -> bool:
         """Check if a filename corresponds to a remote config file.
 
@@ -634,6 +742,62 @@ class RouteMapper:
 #
 # INTERNAL
 #
+def _duckdb_settings(options: Any) -> Tuple[List[str], Optional[int]]:
+    """Turn ``settings.duckdb`` into SET statements and a concurrency cap.
+
+    Args:
+        options: Mapping of DuckDB option -> value, plus optional
+            max_concurrent_queries; or None.
+
+    Returns:
+        Tuple of (SET statements, max concurrent queries or None).
+
+    Raises:
+        ValueError: If options isn't a mapping, an option name isn't a plain
+            identifier, or max_concurrent_queries isn't a positive integer.
+    """
+    if not options:
+        return ([], None)
+    if not isinstance(options, dict):
+        raise ValueError(f"settings.{DUCKDB_SETTINGS_KEY} must be a mapping")
+
+    max_queries = options.get(MAX_CONCURRENT_QUERIES_KEY)
+    if max_queries is not None and (not isinstance(max_queries, int) or max_queries < 1):
+        raise ValueError(f"settings.{DUCKDB_SETTINGS_KEY}.{MAX_CONCURRENT_QUERIES_KEY} "
+                         "must be a positive integer")
+
+    statements = []
+    for name, value in options.items():
+        if name == MAX_CONCURRENT_QUERIES_KEY:
+            continue
+        if not DUCKDB_OPTION_PATTERN.match(str(name)):
+            raise ValueError(f"settings.{DUCKDB_SETTINGS_KEY}: invalid option name '{name}'")
+        if isinstance(value, bool):
+            literal = "true" if value else "false"
+        elif isinstance(value, (int, float)):
+            literal = str(value)
+        else:
+            literal = "'" + str(value).replace("'", "''") + "'"
+        statements.append(f"SET {name} = {literal}")
+    return (statements, max_queries)
+
+
+def _filter_request_headers(headers: Dict[str, str]) -> Dict[str, str]:
+    """Drop request headers that must not be forwarded upstream.
+
+    Args:
+        headers: Incoming client request headers.
+
+    Returns:
+        Headers safe to forward (see EXCLUDED_REQUEST_HEADERS).
+    """
+    return {
+        key: value
+        for key, value in headers.items()
+        if key.lower() not in EXCLUDED_REQUEST_HEADERS
+    }
+
+
 def _resolve_timeout(value: Any) -> Optional[float]:
     """Resolve the configured timeout to seconds, or None to disable it.
 

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: api_dock
-Version: 0.8.1
+Version: 0.8.2
 Summary: A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases
 Author-email: Brookie Guzder-Williams <bguzder-williams@berkeley.edu>
 License-Expression: BSD-3-Clause
@@ -219,19 +219,33 @@ remotes:
 settings:
   add_trailing_slash: true              # Auto-add trailing slash to paths (default: true)
   follow_protocol_downgrades: false     # Allow HTTPS->HTTP redirects (default: false)
+  follow_redirects: true                # Follow remote redirects (default: true)
   timeout: 10                           # Upstream request timeout in seconds (default: 10)
+  base_path: /dock                      # Also serve the API under this prefix (default: none)
+  duckdb:                               # Options for database queries (default: none)
+    memory_limit: 700MB
+    threads: 2
+    max_concurrent_queries: 2
 ```
 
-### HTTP behavior Settings
+### Settings
 
-The optional `settings` section controls HTTP behavior:
+The optional `settings` section controls HTTP and query behavior:
 
 - **`add_trailing_slash`** (default: `true`): Automatically append a trailing slash to all proxied paths. This prevents 307/301 redirects from remote APIs that require trailing slashes (e.g., `/projects` → `/projects/`). Set to `false` to disable this behavior.
 
 - **`follow_protocol_downgrades`** (default: `false`): Control how HTTP redirects are handled. When `false` (recommended), HTTPS→HTTP redirects are blocked for security. When `true`, allows following redirects that downgrade from HTTPS to HTTP (not recommended for production).
 
+- **`follow_redirects`** (default: `true`): Whether redirects from a remote are followed by API Dock (`true`) or passed through to the client with their `Location` header (`false`). Set it to `false` when a remote answers with redirects the client should follow itself, such as presigned S3 URLs for large files.
+
 - **`timeout`** (default: `10`): Upstream request timeout in seconds, applied to both the streaming and buffered proxy paths. Raise it for slow upstreams (e.g. large aggregation queries) that would otherwise return a 502 on timeout. Set to `null` or `false` to disable the timeout entirely (not recommended — a stalled upstream can hold the connection open indefinitely).
 
+- **`base_path`** (default: none): An extra URL prefix the API is also served under, e.g. `/dock`. Use it when a proxy or CDN forwards a path on another domain without stripping it (say CloudFront routes `https://app.example.org/dock/*` to API Dock): `/dock/birdnet/latest/detections/` is then handled as `/birdnet/latest/detections/`. Paths without the prefix keep working, so direct calls and health checks are unaffected.
+
+- **`duckdb`** (default: none): Options for the DuckDB connection each database query runs on. Every key except `max_concurrent_queries` is applied as `SET <key> = <value>`, so any [DuckDB setting](https://duckdb.org/docs/configuration/overview) works; the useful ones on small servers are `memory_limit` (DuckDB spills to disk or fails the query instead of exceeding it), `threads`, and `temp_directory`. `max_concurrent_queries` caps how many database queries run at once in the process; further queries wait their turn. Memory limits apply per query, so on a small instance set `memory_limit × max_concurrent_queries` below the instance's memory.
+
+Database queries run in worker threads, so a slow query doesn't hold up other requests (including health checks on `/`).
+
 ### Catalog Endpoints (`expose`)
 
 The optional `expose` section adds read-only endpoints that list the models and versions of your configured databases, remotes, or both ("sources"). Listings are **opt-in** — with no `expose` key nothing is added.
@@ -1601,7 +1615,7 @@ Publishing a GitHub Release is what publishes to PyPI: `.github/workflows/publis
 
 ```bash
 # 0. Start from a clean, up-to-date main
-export VERSION=0.8.1          # the NEW version, no leading "v"
+export VERSION=0.8.2          # the NEW version, no leading "v"
 git checkout main
 git pull origin main
 git status
@@ -1612,7 +1626,7 @@ git status
 pixi run -e dev pytest -q
 
 # 3. Commit, tag, push (the commit command adds the "v$VERSION: " prefix)
-export COMMIT_MESSAGE='cross-schema unions, schema groups, source columns'
+export COMMIT_MESSAGE='query worker threads, duckdb settings, base_path, proxy host fix'
 git add -A
 git commit -m "v$VERSION: $COMMIT_MESSAGE"
 git tag "v$VERSION"
@@ -1624,17 +1638,14 @@ gh release create "v$VERSION" \
   --title "v$VERSION" \
   --notes "$(cat <<'EOF'
 * new features
-    - Cross-schema unions: `[[*.table]]` reads a table from every shared schema that has it, and `[[*!.table]]` does the same minus the current database/version's schema
-    - Named `schema_groups` in `databases/config.yaml`, used as `[[group.table]]` / `[[group!.table]]` (validated: known schemas only, no group/schema name clashes)
-    - Route `source_columns` adds where each union row came from (`schema`, `name`, `version`), with default names `schema_name`/`name`/`version` or your own; nothing is added by default
-    - `{{self.schema}}`, `{{self.name}}`, and `{{self.version}}` placeholders for the database/version being queried
-    - Together these support an "overlaps" route shared by every database/version (every detection overlapping a given one across all schemas, except that detection itself); shared query params such as `confidence`, `sort`, and `limit` apply to its rows
+    - `settings.duckdb`: DuckDB options applied to every database query (`memory_limit`, `threads`, `temp_directory`, or any other DuckDB setting), plus `max_concurrent_queries` to cap how many queries run at once
+    - `settings.base_path`: also serve the API under a URL prefix (e.g. `/dock`), for a CDN/proxy path such as CloudFront routing `https://app.example.org/dock/*` to API Dock; unprefixed paths keep working
 * bug fixes
-    - A `!` union that removes every member returns no rows (with the right columns) instead of failing, and only the schemas a union actually reads get views and storage credentials
+    - Database queries now run in worker threads, so one slow query no longer stalls every other request (including health checks on `/`)
+    - Remote proxying no longer forwards the client's `Host` (or hop-by-hop) headers upstream; upstream redirects (e.g. trailing-slash 307s) no longer point back at the proxy with the wrong host and path
 * cleanup / other improvements
-    - Added `SqlContext`; `build_sql_query()` / `build_sql_query_with_tables()` accept an optional `context`
-    - README: new "Querying across schemas" section with an overlaps example; example `databases/config.yaml` shows `schema_groups` and a union route
-    - Test suite grew from 198 to 227 tests (`test_schema_unions.py`, including a real DuckDB end-to-end overlaps test)
+    - README: document `base_path`, `duckdb`, and the previously undocumented `follow_redirects` setting
+    - Test suite grew from 227 to 253 tests (`test_runtime_settings.py`)
 EOF
 )"
 

{api_dock-0.8.1 → api_dock-0.8.2}/api_dock.egg-info/SOURCES.txt

@@ -29,6 +29,7 @@ api_dock/example_api_dock_config/remotes/example_remote.yaml
 tests/test_inject_cookies.py
 tests/test_listings.py
 tests/test_proxy_pipeline.py
+tests/test_runtime_settings.py
 tests/test_schema_unions.py
 tests/test_shared_database_config.py
 tests/test_sql_builder.py

{api_dock-0.8.1 → api_dock-0.8.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "api_dock"
-version = "0.8.1"
+version = "0.8.2"
 description = "A flexible API gateway that allows you to proxy requests to multiple remote APIs and Databases"
 readme = "README.md"
 license = "BSD-3-Clause"

api_dock-0.8.2/tests/test_runtime_settings.py

@@ -0,0 +1,249 @@
+"""
+
+Tests for runtime settings and request handling.
+
+Covers ``settings.duckdb`` (per-connection DuckDB options and
+max_concurrent_queries), running database queries off the event loop,
+``settings.base_path`` on the FastAPI and Flask apps, and the request headers
+that are not forwarded to upstream remotes.
+
+License: BSD 3-Clause
+
+"""
+
+#
+# IMPORTS
+#
+import asyncio
+import json
+import time
+from pathlib import Path
+from typing import Any, Dict, List
+from unittest.mock import MagicMock, patch
+
+import pytest
+import yaml
+from fastapi.testclient import TestClient
+
+from api_dock import fast_api, flask_api
+from api_dock.route_mapper import (
+    _duckdb_settings,
+    _filter_request_headers,
+    normalize_base_path,
+    RouteMapper,
+    strip_base_path,
+)
+
+
+#
+# CONSTANTS
+#
+# A deliberately slow, dependency-free query (~0.5-2 s on one thread).
+SLOW_SQL: str = "SELECT SUM(i * i) AS total FROM range(300000000) t(i)"
+
+
+#
+# PUBLIC
+#
+class TestBasePathHelpers:
+    """normalize_base_path / strip_base_path."""
+
+    @pytest.mark.parametrize("value,expected", [
+        (None, None), ("", None), ("/", None),
+        ("dock", "/dock"), ("/dock/", "/dock"), ("/a/b", "/a/b"),
+    ])
+    def test_normalize(self, value: Any, expected: Any) -> None:
+        assert normalize_base_path(value) == expected
+
+    @pytest.mark.parametrize("path,expected", [
+        ("/dock/birdnet/latest/detections/", "/birdnet/latest/detections/"),
+        ("/dock", "/"),
+        ("/dock/", "/"),
+        ("/birdnet/latest/", "/birdnet/latest/"),
+        ("/docks/x", "/docks/x"),
+    ])
+    def test_strip(self, path: str, expected: str) -> None:
+        assert strip_base_path(path, "/dock") == expected
+
+    def test_strip_without_base_path(self) -> None:
+        assert strip_base_path("/dock/x", None) == "/dock/x"
+
+
+class TestDuckdbSettings:
+    """``settings.duckdb`` parsing."""
+
+    def test_statements_and_cap(self) -> None:
+        statements, cap = _duckdb_settings({
+            "memory_limit": "700MB", "threads": 2, "preserve_insertion_order": False,
+            "temp_directory": "/tmp/it's", "max_concurrent_queries": 3,
+        })
+        assert statements == [
+            "SET memory_limit = '700MB'",
+            "SET threads = 2",
+            "SET preserve_insertion_order = false",
+            "SET temp_directory = '/tmp/it''s'",
+        ]
+        assert cap == 3
+
+    def test_empty(self) -> None:
+        assert _duckdb_settings(None) == ([], None)
+
+    @pytest.mark.parametrize("options", [
+        {"bad name": 1}, {"max_concurrent_queries": 0}, {"max_concurrent_queries": "2"}, ["x"],
+    ])
+    def test_invalid(self, options: Any) -> None:
+        with pytest.raises(ValueError):
+            _duckdb_settings(options)
+
+
+class TestRequestHeaderFilter:
+    """Headers not forwarded to remotes."""
+
+    def test_drops_host_and_hop_by_hop(self) -> None:
+        headers = {
+            "Host": "proxy.example.com", "Connection": "keep-alive", "Content-Length": "3",
+            "Accept": "application/json", "Content-Type": "application/json", "X-Custom": "1",
+        }
+        assert _filter_request_headers(headers) == {
+            "Accept": "application/json", "Content-Type": "application/json", "X-Custom": "1",
+        }
+
+    @pytest.mark.anyio
+    async def test_prepared_request_has_no_host(self) -> None:
+        mapper = RouteMapper.__new__(RouteMapper)
+        mapper.remote_names, mapper.database_names, mapper.settings = ["core"], [], {}
+        mapper.config = {"remotes": ["core"]}
+        with patch("api_dock.route_mapper.is_versioned_remote", return_value=False), \
+             patch("api_dock.route_mapper.is_route_allowed", return_value=True), \
+             patch("api_dock.route_mapper.find_remote_config",
+                   return_value={"url": "https://api.example.com"}), \
+             patch("api_dock.route_mapper.filter_remote_query_params",
+                   side_effect=lambda qp, *a, **kw: qp), \
+             patch("api_dock.route_mapper.find_route_mapping", return_value=None), \
+             patch("api_dock.route_mapper.filter_cookies_by_config", return_value={}):
+            prepared = await mapper.prepare_remote_request(
+                "core", "detections/", "GET", headers={"host": "proxy:8000", "accept": "*/*"},
+            )
+        assert prepared.headers == {"accept": "*/*"}
+
+
+class TestQueryExecution:
+    """Database queries: DuckDB options, worker threads, concurrency cap."""
+
+    @pytest.mark.anyio
+    async def test_duckdb_options_applied(
+            self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        mapper = _make_mapper(tmp_path, monkeypatch, settings={
+            "duckdb": {"threads": 1, "memory_limit": "300MB"},
+        })
+        result = await mapper.map_database_route("db", "settings")
+        row = json.loads(result.content)[0]
+        assert row["threads"] == "1"
+        assert row["memory_limit"] != _default_duckdb_setting("memory_limit")
+
+    @pytest.mark.anyio
+    async def test_query_does_not_block_event_loop(
+            self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        mapper = _make_mapper(tmp_path, monkeypatch, settings={"duckdb": {"threads": 1}})
+        gaps: List[float] = []
+
+        async def ticker() -> None:
+            """Record the gap between event-loop wakeups (large = loop was blocked)."""
+            last = time.perf_counter()
+            while True:
+                await asyncio.sleep(0.01)
+                now = time.perf_counter()
+                gaps.append(now - last)
+                last = now
+
+        tick = asyncio.create_task(ticker())
+        start = time.perf_counter()
+        result = await mapper.map_database_route("db", "slow")
+        duration = time.perf_counter() - start
+        tick.cancel()
+
+        assert result.status_code == 200, result.content
+        assert duration > 0.2, 'query too fast to show blocking; raise SLOW_SQL range'
+        assert max(gaps) < duration / 2
+
+    @pytest.mark.anyio
+    async def test_concurrency_cap_acquires_and_releases(
+            self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        mapper = _make_mapper(tmp_path, monkeypatch, settings={
+            "duckdb": {"max_concurrent_queries": 1},
+        })
+        slots = MagicMock()
+        mapper._query_slots = slots
+        await mapper.map_database_route("db", "settings")
+        slots.acquire.assert_called_once()
+        slots.release.assert_called_once()
+
+    @pytest.mark.anyio
+    async def test_failed_query_releases_slot(
+            self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        mapper = _make_mapper(tmp_path, monkeypatch, settings={
+            "duckdb": {"max_concurrent_queries": 1},
+        })
+        result = await mapper.map_database_route("db", "broken")
+        assert result.status_code == 500
+        assert mapper._query_slots.acquire(blocking=False)
+
+
+class TestBasePathApps:
+    """``settings.base_path`` on the FastAPI and Flask apps."""
+
+    def test_fastapi(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        config_path = _write_config(tmp_path, monkeypatch, settings={"base_path": "/dock"})
+        client = TestClient(fast_api.create_app(config_path))
+        for prefix in ("", "/dock"):
+            assert client.get(f"{prefix}/").json()["name"] == "t"
+            assert client.get(f"{prefix}/databases").json() == ["db"]
+            assert client.get(f"{prefix}/db/settings").status_code == 200
+        assert client.get("/docks/db/settings").status_code == 404
+
+    def test_flask(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+        config_path = _write_config(tmp_path, monkeypatch, settings={"base_path": "dock"})
+        client = flask_api.create_app(config_path).test_client()
+        for prefix in ("", "/dock"):
+            assert client.get(f"{prefix}/").get_json()["name"] == "t"
+            assert client.get(f"{prefix}/databases").get_json() == ["db"]
+            assert client.get(f"{prefix}/db/settings").status_code == 200
+
+
+#
+# INTERNAL
+#
+def _write_config(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
+                  settings: Dict[str, Any]) -> str:
+    """Write a one-database config under tmp_path, chdir there, return its path."""
+    config_dir = tmp_path / "api_dock_config"
+    (config_dir / "databases").mkdir(parents=True)
+    (config_dir / "config.yaml").write_text(yaml.safe_dump({
+        "name": "t", "databases": ["db"], "expose": {"databases": True, "dict": False},
+        "settings": settings,
+    }))
+    (config_dir / "databases" / "db.yaml").write_text(yaml.safe_dump({
+        "name": "db",
+        "routes": [
+            {"route": "settings", "sql": "SELECT current_setting('threads')::VARCHAR AS threads, "
+                                         "current_setting('memory_limit') AS memory_limit"},
+            {"route": "slow", "sql": SLOW_SQL},
+            {"route": "broken", "sql": "SELECT * FROM no_such_table"},
+        ],
+    }))
+    monkeypatch.chdir(tmp_path)
+    return str(config_dir / "config.yaml")
+
+
+def _default_duckdb_setting(name: str) -> str:
+    """A DuckDB setting's value on a fresh connection with no options applied."""
+    import duckdb
+
+    with duckdb.connect() as conn:
+        return str(conn.execute(f"SELECT current_setting('{name}')").fetchone()[0])
+
+
+def _make_mapper(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
+                 settings: Dict[str, Any]) -> RouteMapper:
+    """Build a RouteMapper over the one-database test config."""
+    return RouteMapper(_write_config(tmp_path, monkeypatch, settings))