gwdatafind-server 2.2.0__tar.gz → 2.3.0__tar.gz

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/.gitlab-ci.yml

@@ -53,6 +53,7 @@ include:
               --bind=0.0.0.0:8080
               --config=/etc/gwdatafind/gunicorn.conf.py
               --forwarded-allow-ips="*"
+              --workers=1
               "gwdatafind_server:create_app()"
       # deploy options
       deploy_tag: $CI_COMMIT_REF_NAME  # use verbatim tag name

gwdatafind_server-2.3.0/AGENTS.md

@@ -0,0 +1,94 @@
+# AGENTS.md
+
+Guidance for AI coding agents (and humans) working in this repository.
+
+## Project Overview
+
+GWDataFind Server is a [Flask](https://flask.palletsprojects.com/) application that
+implements the HTTP API used to discover URLs for gravitational-wave observatory data
+files.
+It is developed by the LIGO Scientific Collaboration / IGWN and used in production
+to serve data discovery requests for LIGO/Virgo/KAGRA data.
+
+The accompanying Python client library (`gwdatafind`, not in this repo) is what most
+end users interact with; this repo is the server side of that API.
+
+Canonical docs live in `docs/` (built with MkDocs, published at
+<https://computing.docs.ligo.org/gwdatafind/server/>).
+Read `docs/overview.md` first for architecture,
+and `docs/configuration.md` for the full config schema —
+both are kept up to date and should be treated as source of truth over any summary here.
+
+## Architecture (brief)
+
+-   The app (`gwdatafind_server/app.py`, `GWDataFindApp`) runs one or more **sources**
+    (`gwdatafind_server/source/`: `diskcache`, `pelican`) on independent async schedules
+    (via `flask-apscheduler`). Each source discovers fully-qualified URLs
+    (`file://`, `pelican://`, etc.) plus metadata (observatory, dataset, format, GPS
+    segment) and feeds it into an in-memory `Cache` (`gwdatafind_server/cache.py`).
+-   **URL transforms** (`gwdatafind_server/urls.py`, configured under `[url.transform]`)
+    rewrite URLs between schemes (e.g. `pelican://` → `https://`) at query time.
+-   **Auth** (`gwdatafind_server/auth/`) supports SciToken bearer-token authorization and
+    per-IP/subnet overrides.
+-   **Credentials** (`gwdatafind_server/credential/`) are refreshed on a schedule and
+    attached to sources that need them (e.g. SciTokens obtained via Kerberos keytab for
+    Pelican access).
+-   **API** (`gwdatafind_server/api/`): versioned HTTP endpoints (`ldr` legacy, `v1`
+    current) built on the shared cache; see `docs/api/`.
+-   Configuration is TOML (`gwdatafind_server/config.py`), one config file drives the
+    whole app. `python3 -m gwdatafind_server.config <file>` parses and dumps it as JSON
+    for validation.
+
+## Repo layout
+
+-   `gwdatafind_server/` — the application package (source of truth for behavior).
+-   `tests/` — pytest suite, one `test_*.py` per module roughly mirroring the package.
+-   `docs/` — MkDocs site (nav defined in `mkdocs.yml`); includes API reference and
+    deployment examples (`docs/deploy/examples/`).
+-   `examples/` — runnable docker-compose deployment examples (diskcache, auth/SciToken).
+-   `config/` — example/reference config files (TOML, gunicorn, systemd unit) referenced
+    by the docs via `include-markdown`; treat as documentation fixtures, not scratch space.
+-   `Dockerfile`, `docker-compose.yml` — container build/quick-start.
+-   `.gitlab-ci.yml` — CI pipeline (GitLab, not GitHub). Runs lint/type-check/tests across
+    Python 3.11–3.14, builds the Docker image, and builds the MkDocs site.
+
+## Development practices
+
+-   **Dependency management**: [`uv`](https://docs.astral.sh/uv/).
+    Use `uv run …` / `uv sync --group dev` rather than raw `pip`.
+-   **Tests**: `pytest` (`uv run pytest`).
+    Config in `pyproject.toml` (`[tool.pytest.ini_options]`);
+    warnings are errors (`filterwarnings = ["error", ...]`),
+    so new code must not raise unexpected warnings.
+    Coverage is measured against `gwdatafind_server`.
+-   **Linting**: `ruff`, with `select = ["ALL"]` and an explicit ignore list in
+    `pyproject.toml`.
+    Run `uv run ruff check` before considering work done.
+    Per-directory ignores exist for `tests/`, `config/`, `docs/`.
+-   **Type checking**: [`ty`](https://github.com/astral-sh/ty) (not mypy) —
+    see `[tool.ty.src]` in `pyproject.toml`.
+    Note the codebase mostly ignores annotation lint rules (`ANN` is ruff-ignored)
+    but does use `from __future__ import annotations` and `TYPE_CHECKING` blocks.
+-   **Docs**: MkDocs + `mkdocs-material`-based theme, built via a shared CI component.
+    Markdown is linted with `markdownlint` (`.markdownlint.yml`); config docs pull live
+    examples from `config/` via the `include-markdown` plugin, so keep those files valid.
+-   **Commit style**: Conventional-commit-like scoped messages are the norm, e.g.
+    `feat(config): add support for source update jitter`,
+    `fix(pelican): don't skip recursion using cache`,
+    `build(docker-compose): add volume for pelican caches`.
+    Follow this pattern.
+-   **Workflow**: Development happens on GitLab (`git.ligo.org/computing/gwdatafind/server`)
+    via merge requests into `main`;
+    releases are tagged and generate `CHANGELOG.md` entries per version.
+    Licensed GPL-3.0-or-later — keep the copyright/license header convention
+    (`# Copyright (c) 2019-2025 Cardiff University` /
+    `# Licensed under GPLv3+ - see LICENSE`) on new source files.
+
+## Sensitive areas
+
+-   **Auth/credential code** (`gwdatafind_server/auth/`, `gwdatafind_server/credential/`)
+    gates access to (sometimes proprietary/embargoed) LIGO/Virgo/KAGRA data —
+    change carefully and preserve existing test coverage.
+-   **URL transform logic** (`gwdatafind_server/urls.py`) is order-sensitive and affects
+    which URLs get served for which scheme; see the "Transform behavior" warning in
+    `docs/configuration.md` before changing matching/precedence semantics.

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 2.3.0 - 2025-07-06 { #2.3.0 }
+
+-   Load source cache upfront during app startup, use concurrent gunicorn workers
+    ([!133](https://git.ligo.org/computing/gwdatafind/server/-/merge_requests/133))
+-   Improve Pelican sqlite db transaction locking
+    ([!134](https://git.ligo.org/computing/gwdatafind/server/-/merge_requests/134))
+
+[Full details](https://git.ligo.org/computing/gwdatafind/server/-/releases/2.3.0)
+
 ## 2.2.0 - 2025-04-09 { #2.2.0 }
 
 -   Fix directory recursion with Pelican memory cache

gwdatafind_server-2.3.0/CLAUDE.md

@@ -0,0 +1,94 @@
+# AGENTS.md
+
+Guidance for AI coding agents (and humans) working in this repository.
+
+## Project Overview
+
+GWDataFind Server is a [Flask](https://flask.palletsprojects.com/) application that
+implements the HTTP API used to discover URLs for gravitational-wave observatory data
+files.
+It is developed by the LIGO Scientific Collaboration / IGWN and used in production
+to serve data discovery requests for LIGO/Virgo/KAGRA data.
+
+The accompanying Python client library (`gwdatafind`, not in this repo) is what most
+end users interact with; this repo is the server side of that API.
+
+Canonical docs live in `docs/` (built with MkDocs, published at
+<https://computing.docs.ligo.org/gwdatafind/server/>).
+Read `docs/overview.md` first for architecture,
+and `docs/configuration.md` for the full config schema —
+both are kept up to date and should be treated as source of truth over any summary here.
+
+## Architecture (brief)
+
+-   The app (`gwdatafind_server/app.py`, `GWDataFindApp`) runs one or more **sources**
+    (`gwdatafind_server/source/`: `diskcache`, `pelican`) on independent async schedules
+    (via `flask-apscheduler`). Each source discovers fully-qualified URLs
+    (`file://`, `pelican://`, etc.) plus metadata (observatory, dataset, format, GPS
+    segment) and feeds it into an in-memory `Cache` (`gwdatafind_server/cache.py`).
+-   **URL transforms** (`gwdatafind_server/urls.py`, configured under `[url.transform]`)
+    rewrite URLs between schemes (e.g. `pelican://` → `https://`) at query time.
+-   **Auth** (`gwdatafind_server/auth/`) supports SciToken bearer-token authorization and
+    per-IP/subnet overrides.
+-   **Credentials** (`gwdatafind_server/credential/`) are refreshed on a schedule and
+    attached to sources that need them (e.g. SciTokens obtained via Kerberos keytab for
+    Pelican access).
+-   **API** (`gwdatafind_server/api/`): versioned HTTP endpoints (`ldr` legacy, `v1`
+    current) built on the shared cache; see `docs/api/`.
+-   Configuration is TOML (`gwdatafind_server/config.py`), one config file drives the
+    whole app. `python3 -m gwdatafind_server.config <file>` parses and dumps it as JSON
+    for validation.
+
+## Repo layout
+
+-   `gwdatafind_server/` — the application package (source of truth for behavior).
+-   `tests/` — pytest suite, one `test_*.py` per module roughly mirroring the package.
+-   `docs/` — MkDocs site (nav defined in `mkdocs.yml`); includes API reference and
+    deployment examples (`docs/deploy/examples/`).
+-   `examples/` — runnable docker-compose deployment examples (diskcache, auth/SciToken).
+-   `config/` — example/reference config files (TOML, gunicorn, systemd unit) referenced
+    by the docs via `include-markdown`; treat as documentation fixtures, not scratch space.
+-   `Dockerfile`, `docker-compose.yml` — container build/quick-start.
+-   `.gitlab-ci.yml` — CI pipeline (GitLab, not GitHub). Runs lint/type-check/tests across
+    Python 3.11–3.14, builds the Docker image, and builds the MkDocs site.
+
+## Development practices
+
+-   **Dependency management**: [`uv`](https://docs.astral.sh/uv/).
+    Use `uv run …` / `uv sync --group dev` rather than raw `pip`.
+-   **Tests**: `pytest` (`uv run pytest`).
+    Config in `pyproject.toml` (`[tool.pytest.ini_options]`);
+    warnings are errors (`filterwarnings = ["error", ...]`),
+    so new code must not raise unexpected warnings.
+    Coverage is measured against `gwdatafind_server`.
+-   **Linting**: `ruff`, with `select = ["ALL"]` and an explicit ignore list in
+    `pyproject.toml`.
+    Run `uv run ruff check` before considering work done.
+    Per-directory ignores exist for `tests/`, `config/`, `docs/`.
+-   **Type checking**: [`ty`](https://github.com/astral-sh/ty) (not mypy) —
+    see `[tool.ty.src]` in `pyproject.toml`.
+    Note the codebase mostly ignores annotation lint rules (`ANN` is ruff-ignored)
+    but does use `from __future__ import annotations` and `TYPE_CHECKING` blocks.
+-   **Docs**: MkDocs + `mkdocs-material`-based theme, built via a shared CI component.
+    Markdown is linted with `markdownlint` (`.markdownlint.yml`); config docs pull live
+    examples from `config/` via the `include-markdown` plugin, so keep those files valid.
+-   **Commit style**: Conventional-commit-like scoped messages are the norm, e.g.
+    `feat(config): add support for source update jitter`,
+    `fix(pelican): don't skip recursion using cache`,
+    `build(docker-compose): add volume for pelican caches`.
+    Follow this pattern.
+-   **Workflow**: Development happens on GitLab (`git.ligo.org/computing/gwdatafind/server`)
+    via merge requests into `main`;
+    releases are tagged and generate `CHANGELOG.md` entries per version.
+    Licensed GPL-3.0-or-later — keep the copyright/license header convention
+    (`# Copyright (c) 2019-2025 Cardiff University` /
+    `# Licensed under GPLv3+ - see LICENSE`) on new source files.
+
+## Sensitive areas
+
+-   **Auth/credential code** (`gwdatafind_server/auth/`, `gwdatafind_server/credential/`)
+    gates access to (sometimes proprietary/embargoed) LIGO/Virgo/KAGRA data —
+    change carefully and preserve existing test coverage.
+-   **URL transform logic** (`gwdatafind_server/urls.py`) is order-sensitive and affects
+    which URLs get served for which scheme; see the "Transform behavior" warning in
+    `docs/configuration.md` before changing matching/precedence semantics.

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/Dockerfile

@@ -2,18 +2,14 @@
 
 FROM ghcr.io/astral-sh/uv:python3.14-bookworm AS build
 
-ARG SETUPTOOLS_SCM_PRETEND_VERSION
-
 WORKDIR /code
 
-COPY .git .git
 COPY . ./
 
 # Generate a wheel
-RUN if [ -n "$SETUPTOOLS_SCM_PRETEND_VERSION" ]; then \
-      export SETUPTOOLS_SCM_PRETEND_VERSION="$SETUPTOOLS_SCM_PRETEND_VERSION"; \
-    fi \
-    && uv build --wheel
+RUN git config --global safe.directory $(pwd) \
+    && rm -rf docker_dist/*.whl \
+    && uv build --wheel --out-dir docker_dist/
 
 # -- install
 
@@ -28,10 +24,14 @@ RUN uv venv ${VIRTUAL_ENV}
 
 # Install the requirements
 COPY pyproject.toml .
-RUN uv pip install -r pyproject.toml
+RUN uv pip install \
+      -r pyproject.toml \
+      --extra credentials \
+      --extra gunicorn \
+      --extra pelican
 
 # Install the app
-COPY --from=build /code/dist/*.whl /code
+COPY --from=build /code/docker_dist/*.whl /code
 RUN WHEEL=$(ls /code/*.whl) \
     && uv pip install $WHEEL[credentials,gunicorn,pelican]
 

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gwdatafind-server
-Version: 2.2.0
+Version: 2.3.0
 Summary: The GWDataFind data discover server application
 Author-email: Duncan Macleod <duncan.macleod@ligo.org>, Duncan Meacher <duncan.meacher@ligo.org>
 License: GPL-3.0-or-later

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/config/gunicorn.conf.py

@@ -11,20 +11,17 @@ bind = "0.0.0.0:8080"
 backlog = 2048
 
 # Worker processes
-workers = 1
+workers = 2
 worker_class = "gthread"
 worker_connections = 1000
-threads = 8
+threads = 4
 timeout = 30
 keepalive = 2
 
-# Restart workers after this many requests, with up to this much jitter
-max_requests = 1000
-max_requests_jitter = 100
-
-# Worker recycling
-max_worker_memory = 134217728  # 128MB in bytes
-worker_tmp_dir = "/dev/shm"
+# Restart workers after this many requests, with up to this much jitter.
+# Keep this high enough to avoid frequent cache rebuilds.
+max_requests = 10000
+max_requests_jitter = 1000
 
 # Security
 limit_request_line = 4094
@@ -104,6 +101,7 @@ if os.environ.get("GWDATAFIND_SERVER_DEBUG"):
     loglevel = "debug"
     reload = True
     workers = 1
+    threads = 4
 
 if gwdfworkers := os.environ.get("GWDATAFIND_WORKERS"):
     workers = int(gwdfworkers)

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/config/gwdatafind-server.toml

@@ -36,6 +36,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/gwdata"
 interval = 10800
+jitter = 600
 cachefile = "/var/lib/gwdatafind/pelican-gwosc.db"
 
 # -- credentials ------------

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/docker-compose.yml

@@ -10,7 +10,7 @@
 services:
   gwdatafind:
     container_name: gwdatafind-server
-    image: gwdatafind-server:latest
+    image: ${GWDATAFIND_SERVER_IMAGE-containers.ligo.org/computing/gwdatafind/server:latest}
     ports:
       - "127.0.0.1:8080:8080"  # Bind only to localhost
     restart: unless-stopped
@@ -26,13 +26,16 @@ services:
     # Volumes
     volumes:
       - type: bind
-        source: config/gwdatafind-server.toml
+        source: ${GWDATAFIND_SERVER_CONFIG-config/gwdatafind-server.toml}
         target: /etc/gwdatafind/gwdatafind-server.toml
         read_only: true
       - type: tmpfs
         target: /tmp
         tmpfs:
           size: 100m
+      - type: volume
+        source: gwdatafind-data
+        target: /var/lib/gwdatafind
 
     # Environment
     environment:
@@ -41,11 +44,11 @@ services:
 
     # Health check
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8080/api/version", "--max-time", "5"]
+      test: ["CMD-SHELL", "pgrep -f 'gunicorn.*gwdatafind_server:create_app\\(\\)' >/dev/null"]
       interval: 30s
-      timeout: 10s
+      timeout: 5s
       retries: 3
-      start_period: 40s
+      start_period: 30s
 
     # Logging
     logging:
@@ -58,3 +61,7 @@ services:
     labels:
       - "com.example.service=gwdatafind-server"
       - "com.example.version=latest"
+
+volumes:
+  gwdatafind-data:
+    driver: local

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/docs/changelog.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 2.3.0 - 2025-07-06 { #2.3.0 }
+
+-   Load source cache upfront during app startup, use concurrent gunicorn workers
+    ([!133](https://git.ligo.org/computing/gwdatafind/server/-/merge_requests/133))
+-   Improve Pelican sqlite db transaction locking
+    ([!134](https://git.ligo.org/computing/gwdatafind/server/-/merge_requests/134))
+
+[Full details](https://git.ligo.org/computing/gwdatafind/server/-/releases/2.3.0)
+
 ## 2.2.0 - 2025-04-09 { #2.2.0 }
 
 -   Fix directory recursion with Pelican memory cache

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/docs/configuration.md

@@ -257,6 +257,7 @@ All source types accept the following options:
 | `name` | [string](https://toml.io/en/v1.0.0#string) | [optional] A unique name for this source (defaults to "{type}_source_{index}") |
 | `interval` | [integer](https://toml.io/en/v1.0.0#integer) | [optional] How often (in seconds) to query for new data |
 | `initial_delay` | [integer](https://toml.io/en/v1.0.0#integer) | [optional] Delay (in seconds) before the first data collection |
+| `jitter` | [integer](https://toml.io/en/v1.0.0#integer) | [optional] Maximum random jitter (in seconds) to add to the interval between data collections (default: 0) |
 | `credential` | [string](https://toml.io/en/v1.0.0#string) | [optional] Name of the credential to use for authentication |
 | `split_sites` | [boolean](https://toml.io/en/v1.0.0#boolean) | [optional] If `true`, multi-site files will be indexed for individual sites, as well as the combination (default: `false`) |
 
@@ -270,7 +271,6 @@ other keys:
 | `host` | [string](https://toml.io/en/v1.0.0#string) | The address of the diskcache instance |
 | `port` | [integer](https://toml.io/en/v1.0.0#integer) | The port to talk to on the diskcache host |
 | `path` | [string](https://toml.io/en/v1.0.0#string) | Alternatively, the path of a Diskcache ASCII output file to read |
-| `interval` | [integer](https://toml.io/en/v1.0.0#integer) | How often (in seconds) to request information from the diskcache host, or to reread the diskcache file |
 
 !!! example "Diskcache input configuration"
 
@@ -306,7 +306,6 @@ other keys:
 | `token` | [boolean](https://toml.io/en/v1.0.0#boolean) | If `true`, a token will be included in requests to the Pelican host (default `false`) |
 | `credential` | [string](https://toml.io/en/v1.0.0#string) | [optional] Name of the credential to use for authentication (alternative to `token`) |
 | `cachefile` | [string](https://toml.io/en/v1.0.0#string) | [optional] Path to SQLite cache file for persistent storage of directory listings (improves startup performance) |
-| `interval` | [integer](https://toml.io/en/v1.0.0#integer) | How often (in seconds) to query for new data from the Pelican host |
 
 !!! note "Pelican directory listings are cached"
 
@@ -353,7 +352,6 @@ other keys:
 | `principal` | [string](https://toml.io/en/v1.0.0#string) | [optional] The Kerberos principal to authenticate as |
 | `audience` | [string](https://toml.io/en/v1.0.0#string) | The intended audience for the SciToken |
 | `scopes` | [array](https://toml.io/en/v1.0.0#array) of [string](https://toml.io/en/v1.0.0#string) | [optional] List of scopes to request for the SciToken |
-| `interval` | [integer](https://toml.io/en/v1.0.0#integer) | [optional] How often (in seconds) to refresh the credential (default: 3600 = 1 hour) |
 
 !!! example "SciToken credential configuration"
 

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/examples/auth/docker-compose.yml

@@ -10,7 +10,7 @@ services:
       # Rebuild image with Kerberos keytab embedded securely
       context: .
       dockerfile_inline: |
-        FROM containers.ligo.org/computing/gwdatafind/server:latest
+        FROM ${GWDATAFIND_SERVER_IMAGE-containers.ligo.org/computing/gwdatafind/server:latest}
         COPY --chown=gwdatafind:gwdatafind --chmod=400 pelican.keytab /pelican.keytab
     restart: unless-stopped
     ports:
@@ -26,9 +26,12 @@ services:
     # Volumes
     volumes:
       - type: bind
-        source: ./gwdatafind-server.toml
+        source: ${GWDATAFIND_SERVER_CONFIG-./gwdatafind-server.toml}
         target: /etc/gwdatafind/gwdatafind-server.toml
         read_only: true
+      - type: volume
+        source: gwdatafind-data
+        target: /var/lib/gwdatafind
     read_only: true   # Read-only root filesystem
     tmpfs:
       - /tmp:noexec,nosuid,size=100m
@@ -64,3 +67,7 @@ services:
       options:
         max-size: "10m"
         max-file: "3"
+
+volumes:
+  gwdatafind-data:
+    driver: local

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/examples/auth/gwdatafind-server.toml

@@ -38,6 +38,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/gwdata"
 interval = 10800
+jitter = 600
 cachefile = "/var/lib/gwdatafind/pelican-gwosc.db"
 
 # Private LIGO data in Pelican
@@ -47,6 +48,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/igwn/ligo"
 interval = 600
+jitter = 60
 credential = "pelican_token"
 cachefile = "/var/lib/gwdatafind/pelican-ligo.db"
 
@@ -57,6 +59,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/igwn/virgo"
 interval = 600
+jitter = 60
 credential = "pelican_token"
 cachefile = "/var/lib/gwdatafind/pelican-virgo.db"
 
@@ -67,6 +70,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/igwn/kagra"
 interval = 600
+jitter = 60
 credential = "pelican_token"
 cachefile = "/var/lib/gwdatafind/pelican-kagra.db"
 

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/examples/diskcache/docker-compose.yml

@@ -30,6 +30,7 @@ services:
   gwdatafind:
     container_name: gwdatafind-server
     image: containers.ligo.org/computing/gwdatafind/server:latest
+    image: ${GWDATAFIND_SERVER_IMAGE-containers.ligo.org/computing/gwdatafind/server:latest}
     ports:
       - "8080:8080"
     restart: unless-stopped
@@ -45,9 +46,12 @@ services:
     # Volumes
     volumes:
       - type: bind
-        source: gwdatafind-server.toml
+        source: ${GWDATAFIND_SERVER_CONFIG-./gwdatafind-server.toml}
         target: /etc/gwdatafind/gwdatafind-server.toml
         read_only: true
+      - type: volume
+        source: gwdatafind-data
+        target: /var/lib/gwdatafind
     tmpfs:
       - /tmp:noexec,nosuid,size=100m
       - /var/tmp:noexec,nosuid,size=10m
@@ -69,3 +73,8 @@ services:
 networks:
   gwdatafind_internal:
     driver: bridge
+
+# Volumes
+volumes:
+  gwdatafind-data:
+    driver: local

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/examples/diskcache/gwdatafind-server.toml

@@ -37,6 +37,7 @@ type = "pelican"
 federation_url = "osg-htc.org"
 path = "/gwdata"
 interval = 10800
+jitter = 600
 cachefile = "/var/lib/gwdatafind/pelican-gwosc.db"
 
 # Local diskcache for file system scanning

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/gwdatafind_server/__init__.py

@@ -1,13 +1,11 @@
 # Copyright (c) 2019-2025 Cardiff University
 # Licensed under GPLv3+ - see LICENSE
 
-"""The GWDataFind Service provides a metadata-based
-data discover mechanism for data files.
+"""A metadata-based discovery mechanism for GW data files."""
 
-Users provide a few key details as part of an HTTP(S)
-query and receive URLs that indicate the locations
-of the relevant data.
-"""
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
 
 try:
     from ._version import version as __version__
@@ -17,9 +15,17 @@ except ModuleNotFoundError:  # development mode
 from .app import GWDataFindApp
 from .config import get_config_path
 
+if TYPE_CHECKING:
+    from pathlib import Path
+
 
-def create_app(config=None):
+def create_app(
+    name: str = __name__,
+    config: Path | None = None,
+    app_class: type[GWDataFindApp] = GWDataFindApp,
+    **app_kwargs,
+):
     """Create an instance of the application."""
     if config is None:
         config = get_config_path()
-    return GWDataFindApp(__name__, config)
+    return app_class(name, config, **app_kwargs)

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/gwdatafind_server/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '2.2.0'
-__version_tuple__ = version_tuple = (2, 2, 0)
+__version__ = version = '2.3.0'
+__version_tuple__ = version_tuple = (2, 3, 0)
 
-__commit_id__ = commit_id = 'gc287e7f56'
+__commit_id__ = commit_id = 'gffb5a6c3c'

{gwdatafind_server-2.2.0 → gwdatafind_server-2.3.0}/gwdatafind_server/app.py

@@ -63,13 +63,13 @@ class GWDataFindApp(Flask):
         """Initialize the GWDataFind server application."""
         super().__init__(import_name, *args, **kwargs)
 
-        # load our configuration
+        # Load our configuration
         self._init_config(config)
 
-        # finalise logging config
+        # Finalise logging config
         self._init_logger()
 
-        # initialise auth manager
+        # Initialise auth manager
         self.auth = self._init_auth()
 
         # Initialize cache system
@@ -82,15 +82,23 @@ class GWDataFindApp(Flask):
         self._scheduler = APScheduler()
         self._init_scheduler()
 
-        # start credential refresh polling
+        # Start credential refresh polling
+        # NOTE: includes initial credential get
         self._init_credentials()
 
-        # start data source polling
+        # Start data source polling
+        # NOTE: this performs an initial cache update for each source
+        #       before the scheduler starts accepting requests,
+        #       so it may take some time to start up if there are many
+        #       sources or they have long update times.
         self._init_sources()
 
-        # register all API endpoints
+        # Register all API endpoints
         self._register_blueprints()
 
+        # Start the scheduler only after the initial cache warmup completes.
+        self._scheduler.start()
+
         # announce the app is ready
         self.logger.debug(
             "Initialised GWDataFind server with config: %s",
@@ -166,11 +174,8 @@ class GWDataFindApp(Flask):
         # Initialize scheduler with app
         self._scheduler.init_app(self)
 
-        # Start scheduler
-        self._scheduler.start()
-
     def _init_sources(self):
-        """Initialize data sources with scheduled updates."""
+        """Initialize data sources with scheduled updates and warm caches."""
         # get list of sources
         sources: list[SourceConfig] = get_config_value(
             self.config,
@@ -182,14 +187,23 @@ class GWDataFindApp(Flask):
         if not sources:
             self.logger.warning("No sources configured")
 
+        initial_sources: list[Source] = []
         for i, source in enumerate(sources):
             source.setdefault("name", f"{source['type']}_source_{i}")
-            self._init_source(source)
+            initial_sources.append(self._init_source(source))
+
+        # Populate the cache before the scheduler starts accepting requests
+        for source in initial_sources:
+            self.logger.debug(
+                "Running initial cache update for source: %s",
+                source.name,
+            )
+            self._update_source_job(source)
 
     def _init_source(
         self,
         config: SourceConfig,
-    ):
+    ) -> Source:
         """Add a scheduled job to collect data from a source and update the cache."""
         # Look up the specific credential for this source
         credential = None
@@ -223,18 +237,10 @@ class GWDataFindApp(Flask):
             args=[source],
             trigger="interval",
             seconds=config["interval"],
+            jitter=int(config.get("jitter", 0)),
             name=f"Update {source.name} cache",
         )
-
-        # Do immediate initial update
-        self.logger.debug("Scheduling initial update for source: %s", source.name)
-        self._scheduler.add_job(
-            id=f"initial_{source.name}",
-            func=self._update_source_job,
-            args=[source],
-            trigger="date",
-            name=f"Initial update for {source.name}",
-        )
+        return source
 
     def _update_source_job(
         self,