llm-preflight 2.0.0__tar.gz

llm_preflight-2.0.0/.env.example

@@ -0,0 +1,8 @@
+# Copy this file to .env.production beside your benchmark configuration.
+# Fill only the providers you use. Never commit .env.production.
+
+OPENAI_API_KEY=""
+ANTHROPIC_API_KEY=""
+GEMINI_API_KEY=""
+OPENROUTER_API_KEY=""
+XAI_API_KEY=""

llm_preflight-2.0.0/CHANGELOG.md

@@ -0,0 +1,141 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+## 2.0.0 - 2026-07-16
+
+### Changed
+
+- Rename the project and primary PyPI distribution to **LLM Preflight**
+  (`llm-preflight`): a local, cross-provider preflight for a model switch.
+- Make `llm-preflight` the primary command while retaining `llm-bench` and the
+  `llm_bench` Python import namespace as supported compatibility interfaces.
+- Update public documentation, examples, package artifacts, and release
+  automation to use the new product name and primary command.
+
+## 1.2.2 - 2026-07-16
+
+### Fixed
+
+- Keep plain-prompt validation results in model summaries, quality gates, and
+  recommendation ranking so an invalid output can never pass or be recommended.
+- Reject unknown validation keys and support explicit exact-match validation in
+  ordinary and starter configurations.
+- Calculate request and retry-expanded cost limits from every profile case,
+  warmup, prompt override, and output limit.
+- Preserve per-model results when client setup, runtime URL validation, or a
+  request worker fails instead of aborting the benchmark.
+- Keep transient catalogue probe failures retryable; only structured stable
+  incompatibility evidence changes a model's catalogue classification.
+- Measure successful request latency and time-to-first-token per final attempt,
+  excluding retry backoff, and apply the same retry policy to Responses API
+  requests.
+- Gate CI comparisons on configured cost regressions, reject ambiguous custom
+  prompt names and empty `contains` rules, and make numeric-only checks reject
+  explanatory or contradictory output.
+- Keep interactive catalogue comparisons head-to-head with selected approved
+  models; require a distinct paid-run confirmation even after a stray `y` at
+  the stop-mode prompt; preserve discovery deltas when a candidate run fails.
+- Prefer authoritative ready-text catalogue evidence over model-name heuristics
+  and redact Gemini and xAI key formats from all terminal, JSON, candidate-plan,
+  and result output.
+- Preserve transport retries by classifying socket failures by exception type;
+  bootstrap a catalogue snapshot only after a successful first candidate run.
+- Gate CI comparisons on validation-rate regressions as well as latency,
+  request success, and cost; reject legacy built-in test aliases as custom
+  prompt names.
+- Keep `all` in catalogue review to the four inexpensive functional checks,
+  protect invalid-scheme URL errors from credential echoes, and harden local
+  workspace, exact-model-selection, approval-file, and query-encoding edges.
+
+## 1.2.1 - 2026-07-16
+
+### Fixed
+
+- Keep the credential-free `.env.production` template created by `catalog init`
+  identical to the checked-in `.env.example` template.
+
+## 1.2.0 - 2026-07-16
+
+### Added
+
+- Add the local model lifecycle: `catalog init`, `catalog refresh`, and
+  `catalog prepare`, followed by the normal interactive benchmark flow and
+  explicit `models approve` promotion.
+- Add local catalogue snapshots and model-change diffs; retain `watch-new` and
+  `approve-model` as compatibility aliases.
+- Add interactive `--approve-to` promotion, explicit retry-risk acceptance,
+  and candidate-plan `--replace` protection.
+- Classify catalogue entries as ready text benchmarks, text candidates needing
+  one explicit probe, or incompatible generic-text endpoints using provider and
+  OpenRouter capability evidence.
+- Add `catalog probe` and a local, permission-restricted capability ledger that
+  records only safe compatibility evidence, never response text or credentials.
+- Add `--migration-check`: a one-repetition, no-warmup three-case response and
+  basic-contract preflight for comparing a candidate model with an incumbent.
+- Add a custom-contract tutorial and runnable mock examples for JSON extraction,
+  exact intent routing, and required-content validation.
+- Rename default test packs around their user value: quick migration, exact
+  routing, structured output, numeric instruction, and concurrency health.
+  Keep the former selectors as compatibility aliases.
+
+### Fixed
+
+- Refuse a concurrent benchmark targeting the same results directory before it
+  can issue duplicate paid requests.
+- Migrate legacy catalogue snapshots without reporting every model as changed
+  solely because richer capability metadata was introduced.
+
+## 1.0.3 - 2026-07-13
+
+### Changed
+
+- Keep internal agent, roadmap, launch, and release-runbook material out of
+  public source distributions.
+- Restrict source artifacts to runtime code, package metadata, user-facing
+  documentation, and example configuration files.
+
+## 1.0.2 - 2026-07-13
+
+### Added
+
+- Add `llm-bench --init` to create a safe, deterministic no-key mock benchmark
+  without overwriting an existing configuration.
+- Visually separate interactive setup stages and final terminal results,
+  quality-gate, and decision sections.
+- Render `--dry-run` as a readable terminal plan by default; retain JSON output
+  with `--json` for automation.
+- State the qualified recommendation explicitly and show the interactive
+  command after `--init` creates a mock configuration.
+
+### Fixed
+
+- Exclude models that fail any selected test from fastest, cheapest, and
+  best-value recommendations; list them with their failed test instead.
+- Correct smoke-mode documentation: it reduces repetitions and warmups, but
+  does not suppress selected profile-case or load-test expansion.
+
+## 1.0.1 - 2026-07-13
+
+### Fixed
+
+- Handle Ctrl-C cleanly with exit code `130` and without writing artifacts.
+
+## 1.0.0 - 2026-07-13
+
+### Added
+
+- Cross-provider smoke testing, discovery, deterministic validators, reports,
+  pricing checks, retry diagnostics, and CI-oriented controls.
+- A mock-provider quickstart and `--no-save` for no-key and CI workflows.
+- Retry jitter plus nominal and retry-expanded request/cost planning.
+
+### Security
+
+- Redact all custom request-header values from result artifacts and output.
+- Enforce configured cost ceilings only when complete pricing is available.
+
+### Fixed
+
+- Apply CLI `--tests` selections to budget enforcement.
+- Keep the static type-check security gate green.

llm_preflight-2.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 LLM Preflight contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_preflight-2.0.0/MANIFEST.in

@@ -0,0 +1,18 @@
+include .env.example
+include CHANGELOG.md
+include LICENSE
+include README.md
+include SECURITY.md
+include benchmark.auto.example.json
+include benchmark.example.json
+recursive-include examples *.json
+
+exclude AGENTS.md
+exclude CONTRIBUTING.md
+exclude LAUNCH.md
+exclude Makefile
+exclude RELEASING.md
+prune docs
+prune .github
+prune tests
+prune legacy-pypi-shim

llm_preflight-2.0.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: llm-preflight
+Version: 2.0.0
+Summary: Local, cross-provider preflight checks for an LLM model switch
+License-Expression: MIT
+Keywords: llm,preflight,model-switch,validation,openai,anthropic,gemini
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Benchmark
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: coverage>=7; extra == "dev"
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.15; extra == "dev"
+Provides-Extra: audit
+Requires-Dist: bandit>=1.9; extra == "audit"
+Requires-Dist: detect-secrets>=1.5; extra == "audit"
+Requires-Dist: mypy>=1.19; extra == "audit"
+Provides-Extra: release
+Requires-Dist: build>=1.2; extra == "release"
+Requires-Dist: twine>=6; extra == "release"
+Dynamic: license-file
+Dynamic: requires-python
+
+# LLM Preflight
+
+Know whether a model change is safe before it reaches production. LLM Preflight
+runs a small local preflight across providers and compares validated
+output, response speed, tokens, and estimated cost.
+
+It is a local preflight tool—not a hosted evaluation platform, tracing system,
+RAG framework, or public leaderboard. Its results are evidence for your
+account, network, prompts, and validation rules.
+
+Formerly **LLM Speed Bench**. The primary command is `llm-preflight`;
+`llm-bench` remains a supported compatibility command. See the
+[migration guide](docs/migrating-to-llm-preflight.md).
+
+> [!WARNING]
+> Live benchmarks make paid API requests. Start with the no-key demo, preview
+> the plan before a live run, and keep limits and repetitions small.
+
+## Try it in 60 seconds
+
+Create and run a deterministic local benchmark—no API key or network request:
+
+```bash
+llm-preflight --init
+llm-preflight benchmark.json --no-save
+```
+
+From a source checkout:
+
+```bash
+python3 -m llm_bench.cli --init
+python3 -m llm_bench.cli benchmark.json --no-save
+```
+
+`--init` never overwrites an existing config. It creates a mock benchmark so
+you can see the report and exit behavior before making a paid request.
+
+## Use it when
+
+- You are switching models or providers.
+- A provider publishes a new model or changes a `latest` alias.
+- You need to compare your own prompt's validity, latency, and cost.
+- You want local result artifacts instead of a hosted dashboard.
+
+It measures deterministic test validity, end-to-end latency (p50/p95), time to
+first token, throughput when usage is available, token totals, and estimated
+cost. Result files retain request metadata and per-request observations for
+reproducibility.
+
+## First live run
+
+Python 3.10+ is required; there are no runtime dependencies.
+
+```bash
+cp benchmark.example.json benchmark.json
+cp .env.example .env.production
+# Edit benchmark.json and add only the provider keys you use.
+python3 -m llm_bench.cli benchmark.json --dry-run
+python3 -m llm_bench.cli benchmark.json
+```
+
+The CLI reads `.env.production` beside the config without overriding environment
+variables already set by your shell. Use `--no-env-file` or `--env-file PATH`
+when needed. Runs print a terminal report and, unless `--no-save` is used,
+write JSON and Markdown results under `results/`.
+
+Install the command globally in a virtual environment if preferred:
+
+```bash
+python3 -m pip install llm-preflight
+llm-preflight --init
+```
+
+Run `--doctor` and `--dry-run` before the final command. They make no generation
+requests; the final command is the paid work.
+
+## Change a model safely
+
+This is the core workflow. Put your approved model and candidate model in one
+config, then run the small response-and-contract preflight:
+
+```bash
+llm-preflight benchmark.json --migration-check --dry-run
+llm-preflight benchmark.json --migration-check
+```
+
+It sends three short representative cases to each selected model, once each.
+It answers: did the API work, did each response meet the basic contract, and
+how quickly did the provider start and finish responding? It is a cheap
+compatibility check, not a statistical performance conclusion.
+
+When that passes, run the task-specific checks that match your application—for
+example `exact-routing-check` or `structured-output-check`—before approving a
+switch.
+Use [custom contract tests](docs/custom-tests.md) to express the outputs your
+own feature must preserve.
+
+## Choose your path
+
+**I am new and want to see the tool safely.** Start with the
+[Getting started guide](docs/getting-started.md). It uses a no-key local mock
+before any provider request.
+
+**I know the current and candidate model IDs.** Edit one config, run the
+[migration check](#change-a-model-safely), then add a
+[custom contract test](docs/custom-tests.md) for the output your feature must
+preserve. You do not need the catalogue.
+
+**I want to find and review provider releases.** Use the local catalogue
+lifecycle below. It keeps broad provider metadata separate from the small set
+of models you approve for ongoing testing.
+
+```bash
+llm-preflight catalog init
+llm-preflight catalog refresh benchmarks/watch.json
+# If a model is shown as “Needs one probe”, review and confirm a minimal request:
+llm-preflight catalog probe benchmarks/watch.json
+llm-preflight catalog prepare benchmarks/watch.json \
+  --against benchmarks/approved.json --output benchmarks/candidates.json
+llm-preflight benchmarks/candidates.json --interactive \
+  --approve-to benchmarks/approved.json
+```
+
+Refresh reads metadata only. A probe sends one minimal request only for text
+candidates you select and confirm. The interactive benchmark then lets you
+approve passing models explicitly. Follow the complete
+[catalogue tutorial](docs/model-watch.md) for the decision points.
+
+**I am automating an established contract.** Use
+[CI and JSON output](docs/ci.md), with a saved baseline and `--ci` where a
+regression should fail the pipeline.
+
+## Useful commands once you know your path
+
+```bash
+# Inspect configuration, credentials, and model selection without generation.
+llm-preflight benchmark.json --doctor
+llm-preflight benchmark.json --dry-run
+llm-preflight benchmark.json --pricing-check
+
+# Run a reduced live benchmark.
+llm-preflight benchmark.json --smoke
+
+# Run a single ad hoc prompt.
+llm-preflight --quick "Return only valid JSON with a status field." \
+  --models openai:gpt-5.4-mini
+```
+
+For advanced discovery, interactive runs, CI, baselines, replay, and stop
+modes, see [workflows](docs/workflows.md). For models, environment files,
+custom prompts, and provider-specific options, see
+[configuration](docs/configuration.md).
+
+## What makes a comparison useful
+
+- Keep prompts, system instructions, temperature, and output limits fixed.
+- Validate outputs: a fast malformed response is a failed result.
+- Run from the same host; network distance and provider load affect latency.
+- Treat single-user latency and load testing as separate experiments.
+- Prefer dated model IDs over moving aliases.
+
+The CLI distinguishes `API FAIL` (transport, credentials, provider, or request
+failure) from `API OK / TEST FAIL` (a response that fails your validator).
+Recommendations only consider models that pass every selected test.
+
+## Documentation
+
+- [Getting started](docs/getting-started.md) — safe demo, first paid run, and
+  choosing the right workflow.
+- [Workflows](docs/workflows.md) — discovery, smoke mode, CI, replay, matrix,
+  baselines, and request safety.
+- [Configuration](docs/configuration.md) — providers, custom prompts, presets,
+  aliases, and environment overlays.
+- [Custom contract tests](docs/custom-tests.md) — copyable JSON extraction,
+  exact-routing, and content-rule migration tests.
+- [CLI reference](docs/cli-reference.md) — every command-line option, default,
+  and incompatibility.
+- [Interactive mode](docs/interactive.md) — selection, paid-run preview, and
+  live progress.
+- [CI and JSON output](docs/ci.md) — machine-readable output and exit-code
+  gates.
+- [Result JSON schema](docs/result-schema.md) — fields for integrations and
+  saved benchmark evidence.
+- [Model watch and approval](docs/model-watch.md) — discover, compare, and
+  deliberately promote, re-test, and retire provider models.
+- [Troubleshooting](docs/troubleshooting.md) — installation, credentials,
+  catalogue, and benchmark-result recovery.
+- [Tests, pricing, and safety](docs/tests-pricing-safety.md) — built-in tests,
+  validators, pricing confidence, retries, and sensitive data.
+- [Contributing](CONTRIBUTING.md) — development setup and the TDD workflow.
+- [Security](SECURITY.md) — reporting vulnerabilities.
+
+## Contributing and license
+
+Contributions are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Released
+under the [MIT License](LICENSE).

llm_preflight-2.0.0/README.md

@@ -0,0 +1,196 @@
+# LLM Preflight
+
+Know whether a model change is safe before it reaches production. LLM Preflight
+runs a small local preflight across providers and compares validated
+output, response speed, tokens, and estimated cost.
+
+It is a local preflight tool—not a hosted evaluation platform, tracing system,
+RAG framework, or public leaderboard. Its results are evidence for your
+account, network, prompts, and validation rules.
+
+Formerly **LLM Speed Bench**. The primary command is `llm-preflight`;
+`llm-bench` remains a supported compatibility command. See the
+[migration guide](docs/migrating-to-llm-preflight.md).
+
+> [!WARNING]
+> Live benchmarks make paid API requests. Start with the no-key demo, preview
+> the plan before a live run, and keep limits and repetitions small.
+
+## Try it in 60 seconds
+
+Create and run a deterministic local benchmark—no API key or network request:
+
+```bash
+llm-preflight --init
+llm-preflight benchmark.json --no-save
+```
+
+From a source checkout:
+
+```bash
+python3 -m llm_bench.cli --init
+python3 -m llm_bench.cli benchmark.json --no-save
+```
+
+`--init` never overwrites an existing config. It creates a mock benchmark so
+you can see the report and exit behavior before making a paid request.
+
+## Use it when
+
+- You are switching models or providers.
+- A provider publishes a new model or changes a `latest` alias.
+- You need to compare your own prompt's validity, latency, and cost.
+- You want local result artifacts instead of a hosted dashboard.
+
+It measures deterministic test validity, end-to-end latency (p50/p95), time to
+first token, throughput when usage is available, token totals, and estimated
+cost. Result files retain request metadata and per-request observations for
+reproducibility.
+
+## First live run
+
+Python 3.10+ is required; there are no runtime dependencies.
+
+```bash
+cp benchmark.example.json benchmark.json
+cp .env.example .env.production
+# Edit benchmark.json and add only the provider keys you use.
+python3 -m llm_bench.cli benchmark.json --dry-run
+python3 -m llm_bench.cli benchmark.json
+```
+
+The CLI reads `.env.production` beside the config without overriding environment
+variables already set by your shell. Use `--no-env-file` or `--env-file PATH`
+when needed. Runs print a terminal report and, unless `--no-save` is used,
+write JSON and Markdown results under `results/`.
+
+Install the command globally in a virtual environment if preferred:
+
+```bash
+python3 -m pip install llm-preflight
+llm-preflight --init
+```
+
+Run `--doctor` and `--dry-run` before the final command. They make no generation
+requests; the final command is the paid work.
+
+## Change a model safely
+
+This is the core workflow. Put your approved model and candidate model in one
+config, then run the small response-and-contract preflight:
+
+```bash
+llm-preflight benchmark.json --migration-check --dry-run
+llm-preflight benchmark.json --migration-check
+```
+
+It sends three short representative cases to each selected model, once each.
+It answers: did the API work, did each response meet the basic contract, and
+how quickly did the provider start and finish responding? It is a cheap
+compatibility check, not a statistical performance conclusion.
+
+When that passes, run the task-specific checks that match your application—for
+example `exact-routing-check` or `structured-output-check`—before approving a
+switch.
+Use [custom contract tests](docs/custom-tests.md) to express the outputs your
+own feature must preserve.
+
+## Choose your path
+
+**I am new and want to see the tool safely.** Start with the
+[Getting started guide](docs/getting-started.md). It uses a no-key local mock
+before any provider request.
+
+**I know the current and candidate model IDs.** Edit one config, run the
+[migration check](#change-a-model-safely), then add a
+[custom contract test](docs/custom-tests.md) for the output your feature must
+preserve. You do not need the catalogue.
+
+**I want to find and review provider releases.** Use the local catalogue
+lifecycle below. It keeps broad provider metadata separate from the small set
+of models you approve for ongoing testing.
+
+```bash
+llm-preflight catalog init
+llm-preflight catalog refresh benchmarks/watch.json
+# If a model is shown as “Needs one probe”, review and confirm a minimal request:
+llm-preflight catalog probe benchmarks/watch.json
+llm-preflight catalog prepare benchmarks/watch.json \
+  --against benchmarks/approved.json --output benchmarks/candidates.json
+llm-preflight benchmarks/candidates.json --interactive \
+  --approve-to benchmarks/approved.json
+```
+
+Refresh reads metadata only. A probe sends one minimal request only for text
+candidates you select and confirm. The interactive benchmark then lets you
+approve passing models explicitly. Follow the complete
+[catalogue tutorial](docs/model-watch.md) for the decision points.
+
+**I am automating an established contract.** Use
+[CI and JSON output](docs/ci.md), with a saved baseline and `--ci` where a
+regression should fail the pipeline.
+
+## Useful commands once you know your path
+
+```bash
+# Inspect configuration, credentials, and model selection without generation.
+llm-preflight benchmark.json --doctor
+llm-preflight benchmark.json --dry-run
+llm-preflight benchmark.json --pricing-check
+
+# Run a reduced live benchmark.
+llm-preflight benchmark.json --smoke
+
+# Run a single ad hoc prompt.
+llm-preflight --quick "Return only valid JSON with a status field." \
+  --models openai:gpt-5.4-mini
+```
+
+For advanced discovery, interactive runs, CI, baselines, replay, and stop
+modes, see [workflows](docs/workflows.md). For models, environment files,
+custom prompts, and provider-specific options, see
+[configuration](docs/configuration.md).
+
+## What makes a comparison useful
+
+- Keep prompts, system instructions, temperature, and output limits fixed.
+- Validate outputs: a fast malformed response is a failed result.
+- Run from the same host; network distance and provider load affect latency.
+- Treat single-user latency and load testing as separate experiments.
+- Prefer dated model IDs over moving aliases.
+
+The CLI distinguishes `API FAIL` (transport, credentials, provider, or request
+failure) from `API OK / TEST FAIL` (a response that fails your validator).
+Recommendations only consider models that pass every selected test.
+
+## Documentation
+
+- [Getting started](docs/getting-started.md) — safe demo, first paid run, and
+  choosing the right workflow.
+- [Workflows](docs/workflows.md) — discovery, smoke mode, CI, replay, matrix,
+  baselines, and request safety.
+- [Configuration](docs/configuration.md) — providers, custom prompts, presets,
+  aliases, and environment overlays.
+- [Custom contract tests](docs/custom-tests.md) — copyable JSON extraction,
+  exact-routing, and content-rule migration tests.
+- [CLI reference](docs/cli-reference.md) — every command-line option, default,
+  and incompatibility.
+- [Interactive mode](docs/interactive.md) — selection, paid-run preview, and
+  live progress.
+- [CI and JSON output](docs/ci.md) — machine-readable output and exit-code
+  gates.
+- [Result JSON schema](docs/result-schema.md) — fields for integrations and
+  saved benchmark evidence.
+- [Model watch and approval](docs/model-watch.md) — discover, compare, and
+  deliberately promote, re-test, and retire provider models.
+- [Troubleshooting](docs/troubleshooting.md) — installation, credentials,
+  catalogue, and benchmark-result recovery.
+- [Tests, pricing, and safety](docs/tests-pricing-safety.md) — built-in tests,
+  validators, pricing confidence, retries, and sensitive data.
+- [Contributing](CONTRIBUTING.md) — development setup and the TDD workflow.
+- [Security](SECURITY.md) — reporting vulnerabilities.
+
+## Contributing and license
+
+Contributions are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Released
+under the [MIT License](LICENSE).

llm_preflight-2.0.0/SECURITY.md

@@ -0,0 +1,30 @@
+# Security policy
+
+## Reporting a vulnerability
+
+Do not open a public issue for a suspected vulnerability or leaked credential.
+Use GitHub's private vulnerability reporting feature for this repository. If it
+is unavailable, contact the repository maintainers privately.
+
+Include the affected version, reproduction steps, impact, and any suggested
+mitigation. Do not include active API keys.
+
+## Credential and data handling
+
+- API keys belong in environment variables or an uncommitted
+  `.env.production` file.
+- The CLI does not execute `.env.production` as shell code.
+- Catalog output removes custom request headers.
+- The local catalogue capability ledger records model compatibility outcomes and
+  safe request options, not probe response text or credentials; it is created
+  with owner-only permissions where the platform supports them.
+- Result artifacts redact every custom request-header value.
+- Result files can contain prompts, provider errors, model metadata, and
+  optionally full responses. Treat `results/` as potentially sensitive.
+- Treat benchmark JSON as trusted input. It controls outbound HTTP(S)
+  destinations, custom headers, prompts, and provider-specific request fields.
+  Do not run configurations from untrusted sources without reviewing them.
+- Custom provider URLs must use HTTP or HTTPS, include a host, and may not
+  embed credentials.
+- Revoke and rotate a credential immediately if it is committed or included in
+  logs. Removing it from Git history is not sufficient by itself.