llm-benchmark-manager 0.3.0__tar.gz

llm_benchmark_manager-0.3.0/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+All notable user-facing changes are documented here.
+
+## 0.3.0 - 2026-09-25
+
+### Added
+
+- Portable application-data directory selection through `platformdirs`.
+- Reliable AIPerf executable discovery inside the same Python/pipx environment.
+- NVIDIA AIPerf 0.12.0 as a required package dependency.
+- Provider-agnostic bilingual release documentation (English and Hungarian).
+- Public security, contributing and architecture documentation.
+- Capability-aware model lifecycle and exact persisted progress from the v0.2 work.
+- Asymmetric embedding retry with `input_type="query"` only when the provider explicitly requires it.
+
+### Changed
+
+- Package version advanced to 0.3.0 release candidate.
+- Docker installation now relies on normal package dependencies instead of separately installing AIPerf.
+- Direct `BenchmarkService` artifact storage now uses the same portable data-location policy as the runtime.
+
+### Fixed
+
+- Historical run status summaries no longer change when a model receives a different status in a later retest.
+- 404 model invocation failures are represented as `NOT_AVAILABLE` rather than generic model failure.
+- Alternate valid text response shapes are recognized before classifying a response as empty.
+
+## 0.2.0 - 2026-09-24
+
+- Added capability-aware probes, model states, retries, exact progress, REST/MCP progress surfaces and AIPerf baseline eligibility.
+
+## 0.1.0
+
+- Initial standalone provider discovery, SQLite history, CLI, REST, MCP and AIPerf integration.

llm_benchmark_manager-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,37 @@
+# Contributing
+
+Thank you for contributing to LLM Benchmark Manager.
+
+## Development setup
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -e '.[dev]'
+pytest -q
+```
+
+## Change rules
+
+- Keep the benchmark core independent from any particular agent or router.
+- Never hard-code local usernames, host names, home directories or deployment paths.
+- Provider-specific behavior must be narrowly detected; do not apply a provider workaround to unrelated errors.
+- Preserve historical run data. A later retest must not rewrite old run outcomes.
+- Never expose raw credentials through CLI output, REST, MCP, logs, reports or process arguments.
+- Add or update tests for behavior changes and run the full test suite before submitting a change.
+- Keep public documentation in English and Hungarian when a user-facing workflow changes.
+
+## Before submitting
+
+```bash
+pytest -q
+python -m compileall -q src
+python -m build
+python -m pip check
+```
+
+Also scan the change for credentials, local paths, databases and generated artifacts.
+
+## Generated/local files
+
+Do not commit `.venv`, `dist`, `build`, `*.egg-info`, `.pytest_cache`, local databases, `.env` files, credentials or benchmark artifacts.

llm_benchmark_manager-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 LLM Benchmark Manager contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_benchmark_manager-0.3.0/MANIFEST.in

@@ -0,0 +1,7 @@
+include README.md
+include README.hu.md
+include LICENSE
+include SECURITY.md
+include CONTRIBUTING.md
+include CHANGELOG.md
+recursive-include docs *.md

llm_benchmark_manager-0.3.0/PKG-INFO

@@ -0,0 +1,464 @@
+Metadata-Version: 2.4
+Name: llm-benchmark-manager
+Version: 0.3.0
+Summary: Provider-agnostic LLM discovery, stability testing and AIPerf benchmarking
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/telferi/llm-benchmark-manager
+Project-URL: Repository, https://github.com/telferi/llm-benchmark-manager
+Project-URL: Issues, https://github.com/telferi/llm-benchmark-manager/issues
+Project-URL: Changelog, https://github.com/telferi/llm-benchmark-manager/blob/main/CHANGELOG.md
+Keywords: llm,benchmark,aiperf,mcp,openai-compatible
+Classifier: Development Status :: 4 - Beta
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiperf==0.12.0
+Requires-Dist: httpx<1,>=0.28
+Requires-Dist: pydantic<3,>=2.11
+Requires-Dist: fastapi<1,>=0.141
+Requires-Dist: uvicorn<1,>=0.35
+Requires-Dist: fastmcp<5,>=4
+Requires-Dist: PyYAML<7,>=6
+Requires-Dist: keyring<26,>=25
+Requires-Dist: cryptography<51,>=45
+Requires-Dist: platformdirs<5,>=4
+Provides-Extra: dev
+Requires-Dist: pytest<9,>=8; extra == "dev"
+Requires-Dist: build<2,>=1; extra == "dev"
+Dynamic: license-file
+
+# LLM Benchmark Manager
+
+[English](README.md) | [Magyar](README.hu.md)
+
+LLM Benchmark Manager is a standalone, open-source tool for discovering models exposed by an LLM provider, validating that they can actually be called, classifying their capabilities and health, benchmarking compatible text-generation models with NVIDIA AIPerf, and preserving historical evidence in SQLite.
+
+It is intentionally independent of any agent framework, operating-system user, host name, directory layout, or routing system. It can be used manually from a terminal, automated through a REST API, or controlled by an AI/agent through MCP.
+
+## What it does
+
+A normal full run performs the following pipeline:
+
+1. discover the provider model catalog;
+2. create a persistent progress row for every selected model;
+3. detect or infer the model capability;
+4. run a capability-aware smoke probe;
+5. perform bounded stability checks and retry transient provider failures;
+6. run AIPerf only when the model has a compatible benchmark profile;
+7. classify the model (`ACTIVE`, `UNSTABLE`, `NOT_AVAILABLE`, etc.);
+8. store metrics, normalized errors, progress and status history in SQLite.
+
+The tool never deletes historical benchmark evidence when a model changes state later.
+
+## Release status
+
+Current release: **0.3.0**.
+
+The source repository is public on GitHub. Release artifacts are published to PyPI from the GitHub `v*.*.*` release-tag workflow using PyPI Trusted Publishing, so no long-lived PyPI API token is stored in the repository.
+
+## Requirements
+
+- Python 3.11+
+- Internet/network access to the provider being tested
+- Provider credentials for providers that require authentication
+
+**NVIDIA AIPerf 0.12.0 is a required package dependency.** Installing LLM Benchmark Manager installs AIPerf automatically. You do not need to install AIPerf separately.
+
+AIPerf is currently used by the built-in `CHAT_TEXT` baseline profile. Discovery, capability detection and non-text probes still use the same installed application even when no AIPerf benchmark is applicable to a particular model.
+
+## Installation
+
+### From a source checkout
+
+Recommended for the current release candidate:
+
+```bash
+pipx install .
+llmbench --help
+```
+
+For development:
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -e '.[dev]'
+pytest -q
+```
+
+### From a built wheel
+
+```bash
+pipx install dist/llm_benchmark_manager-0.3.0-py3-none-any.whl
+llmbench --help
+```
+
+### Future PyPI installation
+
+After the project is published to PyPI:
+
+```bash
+pipx install llm-benchmark-manager
+```
+
+### Docker
+
+```bash
+docker build -t llm-benchmark-manager:0.3.0 .
+```
+
+Example REST API mode:
+
+```bash
+docker run --rm \
+  --env-file providers.env \
+  -p 8765:8765 \
+  -v llmbench-data:/data \
+  llm-benchmark-manager:0.3.0 serve --host 0.0.0.0 --port 8765
+```
+
+The image stores application data under `/data`. AIPerf is installed automatically because it is a normal project dependency.
+
+## Quick start
+
+Run the interactive interface:
+
+```bash
+llmbench
+```
+
+For a new provider, enter only the provider endpoint URL and API key:
+
+```text
+New provider
+Endpoint URL: https://provider.example.com
+API key: ***************
+```
+
+The provider slug is derived from the endpoint hostname. If that normalized endpoint already exists, the existing provider record is reused rather than duplicated.
+
+For an existing provider the interactive menu supports:
+
+- full retest;
+- `ACTIVE`-only retest;
+- `UNSTABLE`/`FAILED` retest;
+- refresh discovery and test only `NEW` models;
+- test one model;
+- list known models and states.
+
+## Automation CLI
+
+```bash
+llmbench provider list
+llmbench provider discover PROVIDER
+
+llmbench run PROVIDER --mode full
+llmbench run PROVIDER --mode new
+llmbench run PROVIDER --mode active
+llmbench run PROVIDER --mode unstable_failed
+llmbench run --all --mode full
+
+llmbench model test PROVIDER MODEL_ID
+llmbench results RUN_ID
+llmbench history PROVIDER MODEL_ID
+llmbench export RUN_ID --format json --output run.json
+llmbench export RUN_ID --format csv --output run.csv
+```
+
+`PROVIDER` can be the configured provider identifier/slug accepted by the service.
+
+## Provider compatibility
+
+The built-in generic adapter targets OpenAI-compatible provider APIs and uses conventional endpoints such as:
+
+- `/v1/models`
+- `/v1/chat/completions`
+- `/v1/embeddings`
+
+Provider-specific behavior is handled conservatively. For example, asymmetric embedding models that explicitly report that `input_type` is required are retried using `input_type="query"`. Unrelated HTTP 400 errors are not blindly retried with provider-specific parameters.
+
+A provider may expose models in its catalog that are not callable for the current account or endpoint. Those models are classified separately from genuinely broken models.
+
+## Model states
+
+- `NEW` — discovered but not evaluated yet.
+- `ACTIVE` — passed its supported probe and, where applicable, compatible benchmark profile.
+- `UNSTABLE` — usable, but transient failures or partial benchmark failures were observed.
+- `NOT_AVAILABLE` — discovered, but not callable with the current provider/account/endpoint. HTTP 404 invocation responses normally map here.
+- `INCOMPATIBLE` — the attempted generic request shape does not match the model capability.
+- `UNSUPPORTED` — capability is known, but this release has no safe generic probe or benchmark path for it.
+- `FAILED` — a definitive non-transient failure occurred on the correct supported capability path.
+- `DISABLED` — manually excluded.
+- `MISSING` — previously known but no longer returned by discovery.
+
+A later retest does not rewrite the final-status summary of an earlier run.
+
+## Capabilities
+
+Capabilities are stored independently of model health:
+
+- `CHAT_TEXT`
+- `EMBEDDING`
+- `VISION`
+- `MULTIMODAL`
+- `PARSER`
+- `TRANSLATION`
+- `SAFETY`
+- `RERANK`
+- `SPECIAL`
+- `UNKNOWN`
+
+Detection is conservative and may use provider metadata, model-ID heuristics, probe feedback or an explicit manual source. Capability inference alone never marks a model as failed.
+
+## Retry and classification rules
+
+The default error policy distinguishes provider/account availability from model quality:
+
+- `401` / `403` → authentication/provider run issue; not a model failure;
+- `404` → `NOT_AVAILABLE`; no identical retry;
+- `429` → rate limited; bounded retry using 1/2/5/10 seconds or `Retry-After` when supplied;
+- `503` → overloaded; bounded retry using 1/2/5/10 seconds;
+- `500` / `502` / `504` → transient provider error; bounded retry using 1/3 seconds;
+- timeout/network failure → transient; bounded retry using 1/3 seconds;
+- capability-specific `400` mismatch → `INCOMPATIBLE`;
+- deterministic payload failure on the correct request shape → `FAILED`.
+
+A model that succeeds only after retry is classified `UNSTABLE`.
+
+## AIPerf benchmark profile
+
+The built-in `baseline-v1` profile is used for compatible `CHAT_TEXT` models:
+
+- input sequence length: 128 tokens;
+- output sequence length: 128 tokens;
+- requests: 10;
+- concurrency: 1;
+- streaming: enabled;
+- deterministic synthetic seed: 42.
+
+The provider API key is passed to AIPerf through the child-process environment, never as a command-line argument. The temporary AIPerf configuration file is removed after the run.
+
+`EMBEDDING` and other specialized capabilities are not forced through the text-generation AIPerf profile. A successful supported non-text probe may therefore finish with `SKIPPED_UNSUPPORTED_PROFILE` while the model itself is `ACTIVE`.
+
+## Progress and history
+
+Every selected model receives a `run_model_progress` record before execution. This makes progress exact even when many discovered models never reach AIPerf.
+
+Example CLI output:
+
+```text
+[37/82] 45.1%  provider/model-id
+Capability: CHAT_TEXT
+Smoke: PASS
+Stability: PASS
+AIPerf: RUNNING
+```
+
+Progress stages are:
+
+- `PENDING`
+- `CAPABILITY`
+- `SMOKE`
+- `STABILITY`
+- `BENCHMARK`
+- `DONE`
+
+A run is complete when all selected progress rows reach `DONE`.
+
+## Credentials and secret handling
+
+Interactive API keys are never stored in SQLite.
+
+Credential storage order:
+
+1. operating-system keyring when a usable backend exists;
+2. encrypted local credential vault when no usable keyring exists;
+3. environment-variable references for automation and container/system-service deployments.
+
+The encrypted fallback uses a local Fernet master key and vault under the application data directory. The key and vault are protected with restrictive file modes where the operating system supports them. The local OS account remains part of the trust boundary; this fallback is not hardware-backed secret storage.
+
+For ENV-based automation:
+
+```bash
+export PROVIDER_API_KEY='...'
+
+llmbench provider add \
+  --slug example \
+  --name 'Example Provider' \
+  --url 'https://provider.example.com' \
+  --credential-env PROVIDER_API_KEY
+```
+
+Only the environment-variable name is persisted.
+
+Provider errors are sanitized before persistence. Active secrets, bearer tokens, UUID-like request identifiers and account-like identifiers are redacted or normalized, and messages are length-capped.
+
+## Data location
+
+The default data directory is selected with `platformdirs`, so it follows the operating system/user environment instead of a hard-coded home directory.
+
+Typical locations are:
+
+- Linux: `~/.local/share/llmbench/`
+- macOS: `~/Library/Application Support/llmbench/`
+- Windows: the current user's local application-data directory under `llmbench`
+
+Override the location anywhere with:
+
+```bash
+export LLMBENCH_DATA_DIR=/path/to/llmbench-data
+```
+
+The directory contains:
+
+```text
+llmbench.db
+artifacts/
+credentials/     # only when encrypted-file fallback is used
+```
+
+No project code assumes a specific username, server name or installation directory.
+
+## AIPerf executable resolution
+
+AIPerf is a required dependency. LLM Benchmark Manager resolves the executable in this order:
+
+1. `LLMBENCH_AIPERF` explicit override;
+2. `aiperf` installed beside the Python interpreter running `llmbench` (important for `pipx` environments);
+3. `aiperf` found on `PATH`.
+
+Example override:
+
+```bash
+export LLMBENCH_AIPERF=/custom/venv/bin/aiperf
+```
+
+## REST API
+
+Start the API locally:
+
+```bash
+llmbench serve --host 127.0.0.1 --port 8765
+```
+
+Available API resources include:
+
+```text
+GET  /api/v1/providers
+POST /api/v1/providers
+GET  /api/v1/providers/{provider_id}
+POST /api/v1/providers/{provider_id}/discover
+GET  /api/v1/providers/{provider_id}/models
+
+POST /api/v1/runs
+GET  /api/v1/runs/{run_id}
+GET  /api/v1/runs/{run_id}/progress
+POST /api/v1/runs/{run_id}/cancel
+GET  /api/v1/runs/{run_id}/results
+
+GET  /api/v1/models/{model_db_id}
+GET  /api/v1/models/{model_db_id}/history
+```
+
+The REST server has no built-in multi-user authentication in 0.3.0. The default bind address is loopback. Do not expose it to an untrusted network without an authentication/network-control layer such as a trusted reverse proxy or private network.
+
+## MCP server
+
+Start the stdio MCP server:
+
+```bash
+llmbench mcp
+```
+
+The MCP surface includes:
+
+```text
+benchmark_provider_list
+benchmark_provider_get
+benchmark_provider_discover
+benchmark_model_list
+benchmark_model_get
+benchmark_model_history
+benchmark_run_start
+benchmark_run_status
+benchmark_run_progress
+benchmark_run_cancel
+benchmark_run_results
+benchmark_retest_unstable
+benchmark_test_new_models
+```
+
+There is intentionally no MCP tool that returns raw provider credentials.
+
+This allows an external agent (for example a model-management or model-design service) to request discovery and benchmarks without receiving the underlying secrets.
+
+## Exports
+
+Run data can be exported as JSON or CSV:
+
+```bash
+llmbench export RUN_ID --format json --output run.json
+llmbench export RUN_ID --format csv --output run.csv
+```
+
+The SQLite database remains the authoritative local history.
+
+## Security notes
+
+- Provider credentials are excluded from database records and normal responses.
+- AIPerf receives credentials through environment inheritance, not argv.
+- Persisted provider messages are sanitized.
+- Interactive secret input uses hidden terminal input.
+- Provider URLs may not contain embedded username/password credentials.
+- The REST service should remain private unless an external authentication layer is added.
+- Anyone who can read the encrypted fallback master key and vault as the same OS user can decrypt those credentials; use an OS keyring or external secret manager where a stronger boundary is required.
+
+See [SECURITY.md](SECURITY.md) for the disclosure and deployment policy.
+
+## Architecture
+
+The application is deliberately split into independent layers:
+
+```text
+CLI / REST / MCP
+      |
+BenchmarkService
+      |
++-----+------------------+
+|                        |
+Provider adapter       SQLite
+|                        |
+Discovery/probes      history/progress
+|
+AIPerfRunner (CHAT_TEXT baseline)
+```
+
+The benchmark manager is not a model router and does not modify a production routing system. External systems consume its evidence through CLI, REST, MCP or exported data.
+
+See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for details.
+
+## Development and verification
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+pip install -e '.[dev]'
+pytest -q
+python -m build
+python -m pip check
+```
+
+Before publishing a release, also test installation into a clean environment and verify that `llmbench --help` and `aiperf --version` are available from that environment.
+
+## Contributing
+
+Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md). Do not include real API keys, provider account identifiers, local databases or benchmark artifacts containing private data in issues or commits.
+
+## License
+
+MIT. See [LICENSE](LICENSE).