tapbench 0.1.0__tar.gz

tapbench-0.1.0/.gitignore

@@ -0,0 +1,220 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.env.mppdb
+.env.mppdb~
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml

tapbench-0.1.0/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Mario Juric
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

tapbench-0.1.0/PKG-INFO

@@ -0,0 +1,313 @@
+Metadata-Version: 2.4
+Name: tapbench
+Version: 0.1.0
+Summary: A small fixed-rate benchmark client for TAP asynchronous jobs
+Project-URL: Homepage, https://github.com/mjuric/tapbench
+Project-URL: Repository, https://github.com/mjuric/tapbench
+Project-URL: Issues, https://github.com/mjuric/tapbench/issues
+Author-email: Mario Juric <mjuric@uw.edu>
+License-Expression: BSD-3-Clause
+License-File: LICENSE
+Keywords: ADQL,IVOA,TAP,UWS,benchmark
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Astronomy
+Requires-Python: >=3.12
+Requires-Dist: aiohttp<4,>=3.11
+Description-Content-Type: text/markdown
+
+# tapbench
+
+`tapbench` is a small fixed-rate benchmark client for IVOA TAP asynchronous
+jobs. It schedules one ADQL query at an offered rate, follows each admitted UWS
+job to a terminal phase, optionally streams its primary result, and reports
+latency and throughput. It is intentionally not a general-purpose load-testing
+framework.
+
+Python 3.12 or newer is required.
+
+## Install
+
+Install the command from PyPI with either `uv` or `pipx`:
+
+```bash
+uv tool install tapbench
+# or
+pipx install tapbench
+```
+
+From a checkout, [uv](https://docs.astral.sh/uv/) can create the development
+environment and run the command:
+
+```bash
+uv sync
+uv run tapbench --help
+```
+
+The service credentials must already be present in the process environment as
+`TAPBENCH_USER` and `TAPBENCH_PASSWORD`. Both must be nonempty. The command does
+not read dotenv files and does not accept credentials as command-line options.
+
+## Run a benchmark
+
+```bash
+uv run tapbench run \
+  --url https://example.org/tap \
+  --query-file examples/fast_lookup.sql \
+  --rate 20 \
+  --duration 60 \
+  --max-inflight 200 \
+  --poll-interval 0.5 \
+  --no-download-results
+```
+
+`--url` is the TAP base URL; `tapbench` appends `/async`. `--query-file` must
+contain exactly the ADQL query offered at every opportunity. The query itself
+is never printed or stored in JSON; the report identifies it by file name/path
+and SHA-256 digest.
+
+The `run` options are:
+
+| Option | Meaning |
+| --- | --- |
+| `--url URL` | TAP base URL. Only HTTP(S) URLs without credentials, query text, or fragments are accepted. |
+| `--query-file FILE` | Readable, nonempty file containing one ADQL query. |
+| `--rate FLOAT` | Submission opportunities per second; must be positive and finite. |
+| `--duration FLOAT` | Time during which new opportunities are scheduled; must be positive and finite. |
+| `--max-inflight INTEGER` | Maximum admitted operations whose full local lifecycle has not finished; must be at least one. |
+| `--poll-interval FLOAT` | Approximate phase-poll interval, with reproducible small jitter. |
+| `--request-timeout FLOAT` | Timeout for one HTTP operation, including a result-body operation or cleanup request. |
+| `--job-timeout FLOAT` | Time allowed after job creation to observe a terminal UWS phase. |
+| `--download-results`, `--no-download-results` | Stream completed primary results, or stop after observing `COMPLETED`; downloading is the default. |
+| `--keep-jobs` | Do not delete remote UWS jobs after processing. |
+| `--json-output FILE` | Atomically write the detailed `tapbench.run/v1` report. |
+| `--seed INTEGER` | Seed polling jitter for reproducible runs. |
+| `--progress`, `--no-progress` | Enable or suppress live stderr progress; progress is enabled by default. |
+
+Run `uv run tapbench run --help` for the installed command's defaults.
+
+## Fixed-rate behavior
+
+This is an open-loop offered-rate benchmark. Opportunity `k` has an absolute
+deadline of `k / rate` seconds from the monotonic scheduling epoch, beginning
+with opportunity zero at `t=0`. All deadlines strictly before `duration` are
+offered. Thus `--rate 10 --duration 10` creates exactly 100 opportunities even
+when earlier queries are slow. Polls and result requests do not count toward
+the offered rate, and a late event-loop wake-up does not shift later deadlines.
+
+An opportunity is admitted only if an in-flight slot is immediately available.
+Otherwise it is recorded as `local_scheduling_drop`; it is never queued to run
+later. A `--max-inflight` slot is held from submission start through job
+creation, phase handling, optional result download, and attempted cleanup.
+Consequently, slow downloads or cleanup can legitimately cause later drops.
+This bounds local tasks and network work as well as remote nonterminal jobs.
+
+After the scheduling interval, `tapbench` drains the bounded set of admitted
+operations. The job timeout covers waiting for a terminal phase, beginning
+after successful job creation. Result download and cleanup occur outside that
+job deadline and are instead bounded by per-operation request timeouts.
+
+Job creation, job start, phase polling, and remote cleanup treat an explicit
+HTTP 429 response as transient server backpressure through one shared retry
+policy. `tapbench` honors a valid `Retry-After` delay (seconds or HTTP date),
+adds small jitter to avoid synchronized retries, and otherwise uses bounded
+jittered exponential backoff. Creation retries remain within the request
+timeout, start and polling retries remain within the original job deadline, and
+DELETE retries remain within a separate cleanup deadline. Ambiguous POST
+timeouts and transport failures are not retried. Each record and the terminal
+summary report stage-specific rate-limit counts and cumulative wait times.
+
+After observing a terminal phase, `tapbench` retrieves the standard UWS job
+document before cleanup and reads `creationTime`, `startTime`, and `endTime`.
+It reports query queue duration as `startTime - creationTime` and query
+execution duration as `endTime - startTime`. Timing retrieval is best effort:
+missing, malformed, or unavailable fields remain null and do not change the
+job outcome. A service's timestamp precision limits these measurements; a
+whole-second service may report zero for a subsecond query. The queue duration
+also includes any time the job remained PENDING before the client sent RUN, so
+it is not necessarily pure database scheduler wait.
+
+## Results and success
+
+By default, a `COMPLETED` job is successful only after
+`{job}/results/result` has been completely consumed and no HTTP, TAP, or
+VOTable result error was detected. Response-header, first-byte, complete-body,
+and submission-to-download timings are recorded along with the byte count. The
+body is streamed rather than retained in memory.
+
+With `--no-download-results`, a job is successful as soon as creation succeeds
+and `COMPLETED` is observed. No result-list, result-resource, redirect-target,
+or size-discovery request is made, and all result timing and byte fields are
+null. Cleanup failure is reported separately and does not change an otherwise
+successful operation.
+
+Result inspection is deliberately bounded and does not deserialize a complete
+VOTable. `tapbench` retains only a small prefix while streaming and detects a
+`QUERY_STATUS=ERROR` marker when it appears there. An error marker appearing
+only beyond that inspection bound cannot be detected.
+
+Latency summaries ignore missing observations and use nearest-rank p50, p90,
+p95, and p99 values: sort the samples and select observation
+`ceil(percentile * count)`. An unavailable metric is shown as `unavailable`
+and serialized as JSON `null`.
+
+Throughput labels have distinct denominators:
+
+- achieved submission rate is submission starts divided by the configured
+  scheduling duration, or by elapsed scheduler-active time after interruption;
+- job-creation throughput is created jobs divided by the time from the
+  scheduling epoch through the last successful creation response;
+- server execution throughput is completed jobs with valid UWS timing divided
+  by the server work window from the earliest `startTime` to the latest
+  `endTime`; it is independent of client polling and unavailable when the
+  server timestamps produce a zero-width window;
+- result-completion throughput, when downloading, is successful downloads
+  divided by the download window from the first download start through the
+  last completion;
+- download throughput is successfully downloaded bytes divided by the span
+  of that same download window. Download rates are unavailable when there are
+  no successful downloads or the span is zero.
+
+No throughput denominator includes phase-poll sleeps or post-completion cleanup.
+The report shows how many completed jobs supplied valid UWS timing because
+server execution throughput may use fewer samples than the completed count.
+
+## Progress and terminal output
+
+Progress is operational telemetry; final metrics are recomputed from all
+records. It is written to stderr so the final human summary remains on stdout.
+On a terminal, one line refreshes about once per second:
+
+```text
+[schedule  12.0/60.0s] offered=241 started=238 dropped=3 active=87 created=235 completed=141 error=2 aborted=0 success=139 failed=5 polls=812 submission_rate_limited=4 start_rate_limited=2 rate_limited=11 cleanup_rate_limited=3 downloaded=812 MiB
+```
+
+Once scheduling ends, the label changes rather than implying submissions are
+continuing:
+
+```text
+[drain  72.4s] offered=1200 started=1184 dropped=16 active=87 created=1179 completed=1068 error=9 aborted=1 success=1059 failed=26 polls=6312 submission_rate_limited=18 start_rate_limited=9 rate_limited=47 cleanup_rate_limited=12 downloaded=3.1 GiB
+```
+
+The `downloaded` field is omitted in no-download mode, and stage-specific rate
+limit fields are omitted until nonzero. When stderr is not a TTY, searchable
+`tapbench progress:` snapshots are emitted at most every five seconds, plus one
+at the schedule-to-drain transition. `--no-progress` suppresses all snapshots
+but not the final summary.
+
+The final stdout report includes opportunity, submission, terminal, success,
+failure, HTTP, poll, and byte counts; peak in-flight work; the separate
+throughput values; client-observed and UWS server-side latency percentiles; and
+a primary-failure breakdown. Result rows are omitted in no-download mode.
+
+## JSON output and exit status
+
+`--json-output` records sanitized configuration, UTC correlation timestamps,
+monotonic durations, query identity, aggregates, and one structured record for
+every opportunity, including local drops. Its schema id is `tapbench.run/v1`.
+Enums are strings and unavailable observations are null. Output is flushed and
+atomically replaces the destination, so a failed write does not leave a
+partially encoded report. Per-job records retain the three raw UWS timestamp
+strings alongside the derived queue and execution durations.
+
+Exit codes are:
+
+| Code | Meaning |
+| --- | --- |
+| `0` | The benchmark completed. Individual query failures and local drops are measurements, not process failures. |
+| `2` | Invalid CLI/configuration or missing credentials. No benchmark was started. |
+| `3` | Interrupted partial run. |
+| `4` | A fatal run-level client or output error prevented a valid run. |
+
+On the first SIGINT or SIGTERM, scheduling stops immediately, active operations
+are cancelled, and already-created remote jobs receive bounded best-effort
+cleanup unless `--keep-jobs` was selected. Accumulated records are retained,
+the partial summary and requested JSON are produced, and the command exits 3.
+A second interruption may bypass cleanup and output.
+
+## Security
+
+Basic authentication is applied explicitly to every TAP/UWS request. Job,
+phase, error, result, cleanup, and redirect URLs must remain on the configured
+TAP origin; userinfo and cross-origin redirects are rejected. Displayed and
+serialized URLs omit userinfo, query strings, and fragments, and job identifiers
+are redacted. Errors are bounded and scrubbed of credential values. Neither
+credentials, authorization headers, query text, nor complete job URLs belong in
+terminal output or JSON.
+
+Never commit local credential files. In particular, `.env.mppdb` and backup
+variants are local secrets and are ignored by this repository.
+
+## APDB example queries
+
+The queries in [`examples/`](examples/) use table and column names from the
+[published Rubin APDB schema](https://sdm-schemas.lsst.io/apdb.html):
+
+- `fast_lookup.sql` returns at most ten current object summaries;
+- `selective_join.sql` returns at most 25 reliable r-band detections joined to
+  their current object positions;
+- `expensive_small_result.sql` scans and aggregates source fluxes but returns
+  only one row per band.
+
+The SQL files are included in both source and wheel distributions. In an
+installed wheel they reside under the `tapbench/examples/` package directory;
+the repository copies remain the most convenient paths for command-line use.
+
+They intentionally use unqualified table names, as shown by the published APDB
+schema. If a deployment exposes APDB tables only under a TAP schema qualifier,
+make a local copy and qualify both table names for that deployment. `tapbench`
+does not discover or rewrite schemas at runtime.
+
+For a safe live check, first run each example once at a low rate with a single
+in-flight slot and normal cleanup. Then perform a short no-download benchmark:
+
+```bash
+uv run tapbench run \
+  --url https://mppdb.juriclab.org \
+  --query-file examples/fast_lookup.sql \
+  --rate 1 \
+  --duration 3 \
+  --max-inflight 1 \
+  --no-download-results
+```
+
+Do not use `--keep-jobs` for routine validation. Confirm the report contains
+three opportunities and no sensitive URL or credential material.
+
+## Development
+
+The test suite uses a deterministic local fake TAP service; live service
+behavior is not a replacement for those tests. Run all checks with:
+
+```bash
+uv run ruff format --check .
+uv run ruff check .
+uv run mypy src
+uv run pytest
+```
+
+## Releasing
+
+Package versions are derived from Git tags. Releases use clean tags of the
+form `vX.Y.Z`; for example, tag `v0.1.0` produces package version `0.1.0`.
+Untagged commits produce development versions and are never published.
+
+Pushing a release tag builds and validates the wheel and source distribution,
+prints their complete file manifests, and saves them as a workflow artifact.
+Publishing first pauses at the protected `testpypi` environment. After the
+TestPyPI package is verified, production publishing pauses separately at the
+protected `pypi` environment. Review the artifact names, hashes, and manifests
+before approving either environment; the release workflow uses Trusted
+Publishing and stores no PyPI token.
+
+Before the first release, create pending Trusted Publishers on TestPyPI and
+PyPI with project `tapbench`, owner `mjuric`, repository `tapbench`, workflow
+`publish.yml`, and the corresponding `testpypi` or `pypi` environment. Configure
+both GitHub environments with required reviewers so neither upload can begin
+without explicit approval.